Merge "ONOS sfc scripts"
[functest.git] / docs / devguide / index.rst
1 ******************************
2 OPNFV FUNCTEST developer guide
3 ******************************
4
5 .. toctree::
6    :numbered:
7    :maxdepth: 2
8
9
10 ============
11 Introduction
12 ============
13
14 This document describes how feature projects aiming to run functional tests can
15 be integrated into FuncTest framework.
16
17
18 ================================
19 Functest High level architecture
20 ================================
21
22 Functest is project delivering a test container dedicated to OPNFV.
23 It includes the tools, the scripts and the test scenarios.
24
25 Functest can be described as follow::
26
27  +----------------------+
28  |                      |
29  |   +--------------+   |                  +-------------------+
30  |   |              |   |    Public        |                   |
31  |   | Tools        |   +------------------+      OPNFV        |
32  |   | Scripts      |   |                  | System Under Test |
33  |   | Scenarios    |   +------------------+                   |
34  |   |              |   |    Management    |                   |
35  |   +--------------+   |                  +-------------------+
36  |                      |
37  |    Functest Docker   |
38  |                      |
39  +----------------------+
40
41 Functest deals with internal and external test cases.
42 The Internal test cases in Brahmaputra are:
43
44  * vPing_SSH
45  * vPing_userdata
46  * ODL
47  * Tempest
48  * vIMS
49  * Rally
50
51 The external tescases are:
52
53  * Promise
54  * Doctor
55  * Onos
56  * BGPVPN
57
58 see `[2]`_ for details.
59
60 Functest can also be considered as a framework that may include external OPNFV
61 projects.
62 This framework will ease the integration of the feature test suite to the CI.
63
64 ==================
65 How Functest works
66 ==================
67
68 The installation and the launch of the Functest docker image is described in
69 `[1]`_.
70
71 The Functest docker directories are::
72
73  home
74     |
75     `-- opnfv
76         |-- functest
77         |   |-- conf
78         |   |-- data
79         |   `-- results
80         `-- repos
81             |-- bgpvpn
82             |-- doctor
83             |-- functest
84             |-- odl_integration
85             |-- onos
86             |-- ovno
87             |-- promise
88             |-- rally
89             |-- releng
90             `-- vims-test
91
92 ::
93
94  +-----------+-------------------+---------------------------------------------+
95  | Directory | Subdirectory      | Comments                                    |
96  +-----------+-------------------+---------------------------------------------+
97  |           | <project>/conf    | All the useful configuration file(s) for    |
98  |           |                   | <project> the openstack creds are put there |
99  |           |                   | for CI                                      |
100  |           |                   | It is recommended to push it there when     |
101  |           |                   | passing the credentials to container through|
102  |           |                   | the -v option                               |
103  |           +-------------------+---------------------------------------------+
104  |    opnfv  | <project>/data    | Usefull data, images for <projects>         |
105  |           |                   | By default we put a cirros image:           |
106  |           |                   | cirros-0.3.4-x86_64-disk.img                |
107  |           |                   | This image can be used by any projects      |
108  |           +-------------------+---------------------------------------------+
109  |           | <project>/results | Local result directory of project <project> |
110  +-----------+-------------------+---------------------------------------------+
111  |           | bgpvpn            |                                             |
112  |           +-------------------+                                             +
113  | repos     | doctor            |                                             |
114  |           +-------------------+                                             +
115  |           | functest          |                                             |
116  |           +-------------------+                                             +
117  |           | odl_integration   |                                             |
118  |           +-------------------+    Clone of the useful repositories         +
119  |           | onos              |     These repositories may include:         |
120  |           +-------------------+                - tooling                    +
121  |           | promise           |                - scenario                   |
122  |           +-------------------+                - scripts                    +
123  |           | rally             |                                             |
124  |           +-------------------+                                             +
125  |           | releng            |                                             |
126  |           +-------------------+                                             +
127  |           | vims-test         |                                             |
128  |           +-------------------+                                             +
129  |           | <your project>    |                                             |
130  +-----------+-------------------+---------------------------------------------+
131
132 Before running the test suite, you must prepare the environement by running::
133
134  $ /home/opnfv/repos/functest/docker/prepare_env.sh
135
136 By running prepare_env.sh, you build the test environement required by the tests
137 including the retrieval and sourcing of OpenStack credentials.
138 This is an example of the env variables we have in the docker container:
139
140  * HOSTNAME=373f77816eb0
141  * INSTALLER_TYPE=fuel
142  * repos_dir=/home/opnfv/repos
143  * INSTALLER_IP=10.20.0.2
144  * PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
145  * PWD=/home/opnfv
146  * SHLVL=1
147  * HOME=/home/opnfv
148  * NODE_NAME=opnfv-jump-2
149  * creds=/home/opnfv/functest/conf/openstack.creds
150  * _=/usr/bin/env
151
152 The prepare_env.sh will source the credentials, so once run you should have
153 access to the following env variables::
154
155  root@373f77816eb0:~# env|grep OS_
156  OS_REGION_NAME=RegionOne
157  OS_PROJECT_NAME=admin
158  OS_PASSWORD=admin
159  OS_AUTH_STRATEGY=keystone
160  OS_AUTH_URL=http://172.30.10.70:5000/v2.0
161  OS_USERNAME=admin
162  OS_TENANT_NAME=admin
163  OS_ENDPOINT_TYPE=internalURL
164  OS_NO_CACHE=true
165
166
167 Then you may run the test suite by running::
168
169  $ /home/opnfv/repos/functest/docker/run_tests.sh -t <your project>
170
171 see `[2]`_ for details.
172
173
174
175 =========================
176 How to integrate Functest
177 =========================
178
179 The files of the Functest repository you must modify to integrate Functest are:
180
181  * functest/docker/Dockerfile
182  * functest/docker/common.sh
183  * functest/docker/requirements.pip
184  * functest/docker/run_tests.sh
185  * functest/docker/prepare_env.sh
186  * functest/config_funtest.yaml
187
188
189 Dockerfile
190 ==========
191
192 This file lists the repositories to be cloned in the Functest container.
193 The repositories can be internal or external::
194
195  RUN git clone https://gerrit.opnfv.org/gerrit/<your porject> ${repos_dir}/<your project>
196
197
198 Common.sh
199 ==========
200
201 This file can be used to declare the branch and commit  variables of your
202 projects::
203
204  <YOUR_PROJECT>_BRANCH=$(cat $config_file | grep -w <your project>_branch | awk 'END {print $NF}')
205  <YOUR_PROJECT>_COMMIT=$(cat $config_file | grep -w <your project>_commit | awk 'END {print $NF}')
206
207  echo "<YOUR_PROJECT>_BRANCH=${<YOUR_PROJECT>_BRANCH}"
208  echo "<YOUR_PROJECT>_COMMIT=${<YOUR_PROJECT>_COMMIT}"
209
210
211 requirements.pip
212 ================
213
214 This file can be used to preloaded all the needed Python libraries (and avoid
215 that each project does it)
216 The current libraries used in Functest container are::
217
218  # cat requirements.pip
219  pyyaml==3.10
220  gitpython==1.0.1
221  python-neutronclient==2.6.0
222  python-novaclient==2.28.1
223  python-glanceclient==1.1.0
224  python-cinderclient==1.4.0
225  python-ceilometerclient==1.5.1
226  python-keystoneclient==1.6.0
227  virtualenv==1.11.4
228  pexpect==4.0
229  requests==2.8.0
230  robotframework==2.9.1
231  robotframework-requests==0.3.8
232  robotframework-sshlibrary==2.1.1
233  configObj==5.0.6
234  Flask==0.10.1
235  xmltodict==0.9.2
236  scp==0.10.2
237  paramiko==1.16.0
238
239
240 prepare_env.sh
241 ==============
242
243 This script can be adapted if you need to set up a specific environment before
244 running the tests.
245
246 run_tests.sh
247 ============
248
249 This script is used to run the tests. You must thus complete the cases with your
250 own project::
251
252  ;;
253     "promise")
254         info "Running PROMISE test case..."
255         # TODO
256     ;;
257     "doctor")
258         info "Running Doctor test..."
259         python ${FUNCTEST_REPO_DIR}/testcases/features/doctor.py
260     ;;
261     "<your project>")
262         info "Running <your project> test..."
263         # your script that launchs your tests...
264     ;;
265
266 And do not forget to update also the help line::
267
268  -t|--test         run specific set of tests
269       <test_name>     one or more of the following separated by comma:
270                          vping_ssh,vping_userdata,odl,rally,tempest,vims,onos,promise,ovno
271
272
273 config_funtest.yaml
274 ===================
275
276 This file is the main configuration file of Functest. You must add the
277 references to your project::
278
279  general:
280     directories:
281         dir_repo_<your project>:      /home/opnfv/repos/<your project>
282      repositories:
283         # branch and commit ID to which the repos will be reset (HEAD)
284         <your project>_branch:   master
285         <your project>_commit:   latest
286
287
288 ====================
289 Test Dashboard & API
290 ====================
291
292 The OPNFV testing group created a test collection database to collect the test
293 results from CI.
294 Any test project running on any lab integrated in CI can push the results to
295 this database.
296 This database can be used afterwards to see the evolution of the tests and
297 compare the results versus the installers, the scenario or the labs.
298
299 You can find more information about the dashboard from Testing Dashboard wiki
300 page `[3]`_.
301
302 Overall Architecture
303 ====================
304 The Test result management can be summarized as follows::
305
306   +-------------+    +-------------+    +-------------+
307   |             |    |             |    |             |
308   |   Test      |    |   Test      |    |   Test      |
309   | Project #1  |    | Project #2  |    | Project #N  |
310   |             |    |             |    |             |
311   +-------------+    +-------------+    +-------------+
312            |               |               |
313            V               V               V
314        +-----------------------------------------+
315        |                                         |
316        |         Test Rest API front end         |
317        |  http://testresults.opnfv.org/test      |
318        |                                         |
319        +-----------------------------------------+
320            A                |
321            |                V
322            |     +-------------------------+
323            |     |                         |
324            |     |    Test Results DB      |
325            |     |         Mongo DB        |
326            |     |                         |
327            |     +-------------------------+
328            |
329            |
330      +----------------------+
331      |                      |
332      |    test Dashboard    |
333      |                      |
334      +----------------------+
335
336 Test API description
337 ====================
338 The Test API is used to declare pods, projects, test cases and test results. An
339 additional method dashboard has been added to post-process the raw results in release Brahmaputra. 
340 The data model is very basic, 4 objects are created:
341
342   * Pods
343   * Projects
344   * Testcases
345   * Results
346
347 Pods::
348
349   {
350     "id": <ID>,
351     "details": <URL description of the POD>,
352     "creation_date": YYYY-MM-DD HH:MM:SS ,
353     "name": <The POD Name>,
354     "mode": <metal or virtual>,
355     "role": <ci-pod or community-pod or single-node>
356   },
357
358 Projects::
359
360   {
361     "id": <ID>,
362     "name": <Name of the Project>,
363     "creation_date": "YYYY-MM-DD HH:MM:SS",
364     "description": <Short description>
365   },
366
367 Testcases::
368
369   {
370     "id": <ID>,
371     "name":<Name of the test case>,
372     "project_name":<Name of belonged project>,
373     "creation_date": "YYYY-MM-DD HH:MM:SS",
374     "description": <short description>,
375     "url":<URL for longer description>
376   },
377
378 Results::
379
380   {
381     "_id": <ID,
382     "case_name": <Reference to the test case>
383     "project_name": <Reference to project>,
384     "pod_name": <Reference to POD where the test was executed>,
385     "installer": <Installer Apex or Compass or Fuel or Joid>,
386     "version": <master or Colorado or Brahmaputra>,
387     "start_date": "YYYY-MM-DD HH:MM:SS",
388     "stop_date": "YYYY-MM-DD HH:MM:SS",
389     "build_tag": <such as "jenkins-functest-fuel-baremetal-daily-master-108">,
390     "scenario": <Scenario on which the test was executed>,
391     "criteria": <PASS or FAILED>,
392     "trust_indicator": <0 ~ 1>
393
394 The API can described as follows. For detailed information, please go to
395 http://testresults.opnfv.org/test/swagger/spec.html
396
397 Authentication: opnfv/api@opnfv
398
399 Please notes that POST/DELETE/PUT operations for test or study purpose via 
400 swagger website is not allowed, because it will change the real data in
401 the database.
402
403 Version:
404
405  +--------+--------------------------+-----------------------------------------+
406  | Method | Path                     | Description                             |
407  +========+==========================+=========================================+
408  | GET    | /versions                 | Get all supported API versions         |
409  +--------+--------------------------+-----------------------------------------+
410
411
412 Pods:
413
414  +--------+----------------------------+-----------------------------------------+
415  | Method | Path                       | Description                             |
416  +========+============================+=========================================+
417  | GET    | /api/v1/pods               | Get the list of declared Labs (PODs)    |
418  +--------+----------------------------+-----------------------------------------+
419  | POST   | /api/v1/pods               | Declare a new POD                       |
420  |        |                            | Content-Type: application/json          |
421  |        |                            | {                                       |
422  |        |                            | "name": "pod_foo",                      |
423  |        |                            | "mode": "metal",                        |
424  |        |                            | "role": "ci-pod",                       |
425  |        |                            | "details": "it is a ci pod"             |
426  |        |                            | }                                       |
427  +--------+----------------------------+-----------------------------------------+
428  | GET    | /api/v1/pods/{pod_name}    | Get a declared POD                      |
429  +--------+----------------------------+-----------------------------------------+
430
431 Projects:
432
433  +--------+----------------------------+-----------------------------------------+
434  | Method | Path                       | Description                             |
435  +========+============================+=========================================+
436  | GET    | /api/v1/projects           | Get the list of declared projects       |
437  +--------+----------------------------+-----------------------------------------+
438  | POST   | /api/v1/projects           | Declare a new test project              |
439  |        |                            | Content-Type: application/json          |
440  |        |                            | {                                       |
441  |        |                            | "name": "project_foo",                  |
442  |        |                            | "description": "whatever you want"      |
443  |        |                            | }                                       |
444  +--------+----------------------------+-----------------------------------------+
445  | DELETE | /api/v1/projects/{project} | Delete a test project                   |
446  +--------+----------------------------+-----------------------------------------+
447  | GET    | /api/v1/projects/{project} | Get details on a {project}              |
448  |        |                            |                                         |
449  +--------+----------------------------+-----------------------------------------+
450  | PUT    | /api/v1/projects/{project} | Update a test project                   |
451  |        |                            |                                         |
452  |        |                            | Content-Type: application/json          |
453  |        |                            | {                                       |
454  |        |                            | <the field(s) you want to modify>       |
455  |        |                            | }                                       |
456  +--------+----------------------------+-----------------------------------------+
457
458
459 Testcases:
460
461  +--------+----------------------------+-----------------------------------------+
462  | Method | Path                       | Description                             |
463  +========+============================+=========================================+
464  | GET    | /api/v1/projects/{project}/| Get the list of testcases of {project}  |
465  |        | cases                      |                                         |
466  +--------+----------------------------+-----------------------------------------+
467  | POST   | /api/v1/projects/{project}/| Add a new test case to {project}        |
468  |        | cases                      | Content-Type: application/json          |
469  |        |                            | {                                       |
470  |        |                            | "name": "case_foo",                     |
471  |        |                            | "description": "whatever you want"      |
472  |        |                            | "url": "whatever you want"              |
473  |        |                            | }                                       |
474  +--------+----------------------------+-----------------------------------------+
475  | DELETE | /api/v1/projects/{project}/| Delete a test case                      |
476  |        | cases/{case}               |                                         |
477  +--------+----------------------------+-----------------------------------------+
478  | GET    | /api/v1/projects/{project}/| Get a declared test case                |
479  |        | cases/{case}               |                                         |
480  +--------+----------------------------+-----------------------------------------+
481  | PUT    | /api/v1/projects/{project}?| Modify a test case of {project}         |
482  |        | cases/{case}               |                                         |
483  |        |                            | Content-Type: application/json          |
484  |        |                            | {                                       |
485  |        |                            | <the field(s) you want to modify>       |
486  |        |                            | }                                       |
487  +--------+----------------------------+-----------------------------------------+
488
489 Results:
490
491  +--------+----------------------------+------------------------------------------+
492  | Method | Path                       | Description                              |
493  +========+============================+==========================================+
494  | GET    | /api/v1/results             | Get all the test results                |
495  +--------+----------------------------+------------------------------------------+
496  | POST   | /api/v1/results            | Add a new test results                   |
497  |        |                            | Content-Type: application/json           |
498  |        |                            | {                                        |
499  |        |                            | "project_name": "project_foo",           |
500  |        |                            | "scenario": "odl-l2",                    |
501  |        |                            | "stop_date": "2016-05-28T14:42:58.384Z", |
502  |        |                            | "trust_indicator": 0.5,                  |
503  |        |                            | "case_name": "vPing",                    |
504  |        |                            | "build_tag": "",                         |
505  |        |                            | "version": "Colorado",                   |
506  |        |                            | "pod_name": "pod_foo",                   |
507  |        |                            | "criteria": "PASS",                      |
508  |        |                            | "installer": "fuel",                     |
509  |        |                            | "start_date": "2016-05-28T14:41:58.384Z",|
510  |        |                            | "details": <your results>                |
511  |        |                            | }                                        |
512  +--------+----------------------------+------------------------------------------+
513  | GET    | /api/v1/results?           | Get the test results of {case}           |
514  |        | case={case}                |                                          |
515  +--------+----------------------------+------------------------------------------+
516  | GET    | /api/v1/results?           | Get the test results of build_tag        |
517  |        | build_tag={tag}            | {tag}.                                   |
518  +--------+----------------------------+------------------------------------------+
519  | GET    | /api/v1/results?           | Get last {N} records of test results     |
520  |        | last={N}                   |                                          |
521  +--------+----------------------------+------------------------------------------+
522  | GET    | /api/v1/results?           | Get the test results of scenario         |
523  |        | scenario={scenario}        | {scenario}.                              |
524  +--------+----------------------------+------------------------------------------+
525  | GET    | /api/v1/results?           | Get the test results of trust_indicator  |
526  |        | trust_indicator={ind}      | {ind}.                                   |
527  +--------+----------------------------+------------------------------------------+
528  | GET    | /api/v1/results?           | Get the test results of last days        |
529  |        | period={period}            | {period}.                                |
530  +--------+----------------------------+------------------------------------------+
531  | GET    | /api/v1/results?           | Get the test results of {project}        |
532  |        | project={project}          |                                          |
533  +--------+----------------------------+------------------------------------------+
534  | GET    | /api/v1/results?           | Get the test results of version          |
535  |        | version={version}          | {version}.                               |
536  +--------+----------------------------+------------------------------------------+
537  | GET    | /api/v1/results?           | Get the test results of criteria         |
538  |        | criteria={criteria}        | {criteria}.                              |
539  +--------+----------------------------+------------------------------------------+
540  | GET    | /api/v1/results?           | get the results on pod {pod}             |
541  |        | pod={pod}                  |                                          |
542  +--------+----------------------------+------------------------------------------+
543  | GET    | /api/v1/results?           | Get the test results of installer {inst} |
544  |        | installer={inst}           |                                          |
545  +--------+----------------------------+------------------------------------------+
546  | GET    | /api/v1/results?           | Get the results according to combined    |
547  |        | <query conditions>         | query conditions supported above         |
548  +--------+----------------------------+------------------------------------------+
549  | GET    | /api/v1/results/{result_id}| Get the test result by result_id         |
550  +--------+----------------------------+------------------------------------------+
551
552
553 Dashboard:
554
555  +--------+----------------------------+-----------------------------------------+
556  | Method | Path                       | Description                             |
557  +========+============================+=========================================+
558  | GET    |/dashboard/v1/results?      | Get all the dashboard ready results of  |
559  |        |&project={project}          | {case} of the project {project}         |
560  |        |&case={case}                |                                         |
561  +--------+----------------------------+-----------------------------------------+
562  | GET    |/dashboard/v1/results?      | Get all the dashboard ready results of  |
563  |        |&project={project}          | {case} of the project {project}         |
564  |        |&case={case}                | version {version}                       |
565  |        |&version={version}          |                                         |
566  +--------+----------------------------+-----------------------------------------+
567  | GET    |/dashboard/v1/results?      | Get all the dashboard ready results of  |
568  |        |&project={project}          | {case} of the project {project}         |
569  |        |&case={case}                | since {days} days                       |
570  |        |&period={days}              |                                         |
571  +--------+----------------------------+-----------------------------------------+
572  | GET    |/dashboard/v1/results?      | Get all the dashboard ready results of  |
573  |        |&project={project}          | {case} of the project {project}         |
574  |        |&case={case}                | installed by {installer}                |
575  |        |&installer={installer}      |                                         |
576  +--------+----------------------------+-----------------------------------------+
577  | GET    |/dashboard/v1/results?      | Get all the dashboard ready results of  |
578  |        |&project={project}          | {case} of the project {project}         |
579  |        |&case={case}                | on POD {pod}                            |
580  |        |&pod={pod}                  |                                         |
581  +--------+----------------------------+-----------------------------------------+
582  | GET    |/dashboard/v1/results?      | Get all the dashboard ready results of  |
583  |        |&project={project}          | {case} of the project {project}         |
584  |        |&case={case}                | and combined by other query conditions  |
585  |        |&<query conditions>         | supported above.                        |
586  +--------+----------------------------+-----------------------------------------+
587  | GET    |/dashboard/v1/projects?     | Get all the dashboard ready projects    |
588  +--------+----------------------------+-----------------------------------------+
589
590 Dashboard description
591 =====================
592
593 The results with dashboard method are post-processed from raw results.
594 Please note that dashboard results are not stored. Only raw results are stored.
595
596 Release Brahmaputra
597 -------------------
598
599 Dashboard url: http://testresults.opnfv.org/dashboard/
600
601 Release Colorado
602 ----------------
603
604 Dashboard url: http://testresults.opnfv.org/kibana_dashboards/
605
606 Credentials for a guest account: opnfvuser/kibana
607
608 ===============================================
609 How to push your results into the Test Database
610 ===============================================
611
612 The test database is used to collect test results. By default it is enabled only
613 in Continuous Integration.
614 The architecture and associated API is described in `[2]`_.
615 If you want to push your results from CI, you just have to use the API at the
616 end of your script.
617
618 You can also reuse a python function defined in functest_utils.py::
619
620   def push_results_to_db(db_url, case_name, logger, pod_name,version, payload):
621     """
622     POST results to the Result target DB
623     """
624     url = db_url + "/results"
625     installer = get_installer_type(logger)
626     params = {"project_name": "functest", "case_name": case_name,
627               "pod_name": pod_name, "installer": installer,
628               "version": version, "details": payload}
629
630     headers = {'Content-Type': 'application/json'}
631     try:
632         r = requests.post(url, data=json.dumps(params), headers=headers)
633         if logger:
634             logger.debug(r)
635         return True
636     except Exception, e:
637         print "Error [push_results_to_db('%s', '%s', '%s', '%s', '%s')]:" \
638             % (db_url, case_name, pod_name, version, payload), e
639         return False
640
641
642 ==========
643 References
644 ==========
645
646 .. _`[1]`: http://artifacts.opnfv.org/functest/docs/configguide/index.html Functest configuration guide URL
647 .. _`[2]`: http://artifacts.opnfv.org/functest/docs/userguide/index.html functest user guide URL
648 .. _`[3]`: https://wiki.opnfv.org/opnfv_test_dashboard
649
650
651 OPNFV main site: opnfvmain_.
652
653 OPNFV functional test page: opnfvfunctest_.
654
655 IRC support chan: #opnfv-testperf
656
657 .. _opnfvmain: http://www.opnfv.org
658 .. _opnfvfunctest: https://wiki.opnfv.org/opnfv_functional_testing
659 .. _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html
660 .. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html
661 .. _`config_test.py` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.py
662 .. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml
663