docs/devguide/index.rst

   1 ******************************
   2 OPNFV FUNCTEST developer guide
   3 ******************************
   4
   5 .. toctree::
   6    :numbered:
   7    :maxdepth: 2
   8
   9
  10 ============
  11 Introduction
  12 ============
  13
  14 This document describes how feature projects aiming to run functional tests can
  15 be integrated into FuncTest framework.
  16
  17
  18 ================================
  19 Functest High level architecture
  20 ================================
  21
  22 Functest is project delivering a test container dedicated to OPNFV.
  23 It includes the tools, the scripts and the test scenarios.
  24
  25 Functest can be described as follow::
  26
  27  +----------------------+
  28  |                      |
  29  |   +--------------+   |                  +-------------------+
  30  |   |              |   |    Public        |                   |
  31  |   | Tools        |   +------------------+      OPNFV        |
  32  |   | Scripts      |   |                  | System Under Test |
  33  |   | Scenarios    |   +------------------+                   |
  34  |   |              |   |    Management    |                   |
  35  |   +--------------+   |                  +-------------------+
  36  |                      |
  37  |    Functest Docker   |
  38  |                      |
  39  +----------------------+
  40
  41 Functest deals with internal and external test cases.
  42 The Internal test cases in Brahmaputra are:
  43
  44  * vPing_SSH
  45  * vPing_userdata
  46  * ODL
  47  * Tempest
  48  * vIMS
  49  * Rally
  50
  51 The external tescases are:
  52
  53  * Promise
  54  * Doctor
  55  * Onos
  56  * BGPVPN
  57
  58 see `[2]`_ for details.
  59
  60 Functest can also be considered as a framework that may include external OPNFV
  61 projects.
  62 This framework will ease the integration of the feature test suite to the CI.
  63
  64 ==================
  65 How Functest works
  66 ==================
  67
  68 The installation and the launch of the Functest docker image is described in
  69 `[1]`_.
  70
  71 The Functest docker directories are::
  72
  73  home
  74     |
  75     `-- opnfv
  76         |-- functest
  77         |   |-- conf
  78         |   |-- data
  79         |   `-- results
  80         `-- repos
  81             |-- bgpvpn
  82             |-- doctor
  83             |-- functest
  84             |-- odl_integration
  85             |-- onos
  86             |-- ovno
  87             |-- promise
  88             |-- rally
  89             |-- releng
  90             `-- vims-test
  91
  92 ::
  93
  94  +-----------+-------------------+---------------------------------------------+
  95  | Directory | Subdirectory      | Comments                                    |
  96  +-----------+-------------------+---------------------------------------------+
  97  |           | <project>/conf    | All the useful configuration file(s) for    |
  98  |           |                   | <project> the openstack creds are put there |
  99  |           |                   | for CI                                      |
 100  |           |                   | It is recommended to push it there when     |
 101  |           |                   | passing the credentials to container through|
 102  |           |                   | the -v option                               |
 103  |           +-------------------+---------------------------------------------+
 104  |    opnfv  | <project>/data    | Usefull data, images for <projects>         |
 105  |           |                   | By default we put a cirros image:           |
 106  |           |                   | cirros-0.3.4-x86_64-disk.img                |
 107  |           |                   | This image can be used by any projects      |
 108  |           +-------------------+---------------------------------------------+
 109  |           | <project>/results | Local result directory of project <project> |
 110  +-----------+-------------------+---------------------------------------------+
 111  |           | bgpvpn            |                                             |
 112  |           +-------------------+                                             +
 113  | repos     | doctor            |                                             |
 114  |           +-------------------+                                             +
 115  |           | functest          |                                             |
 116  |           +-------------------+                                             +
 117  |           | odl_integration   |                                             |
 118  |           +-------------------+    Clone of the useful repositories         +
 119  |           | onos              |     These repositories may include:         |
 120  |           +-------------------+                - tooling                    +
 121  |           | promise           |                - scenario                   |
 122  |           +-------------------+                - scripts                    +
 123  |           | rally             |                                             |
 124  |           +-------------------+                                             +
 125  |           | releng            |                                             |
 126  |           +-------------------+                                             +
 127  |           | vims-test         |                                             |
 128  |           +-------------------+                                             +
 129  |           | <your project>    |                                             |
 130  +-----------+-------------------+---------------------------------------------+
 131
 132 Before running the test suite, you must prepare the environement by running::
 133
 134  $ /home/opnfv/repos/functest/docker/prepare_env.sh
 135
 136 By running prepare_env.sh, you build the test environement required by the tests
 137 including the retrieval and sourcing of OpenStack credentials.
 138 This is an example of the env variables we have in the docker container:
 139
 140  * HOSTNAME=373f77816eb0
 141  * INSTALLER_TYPE=fuel
 142  * repos_dir=/home/opnfv/repos
 143  * INSTALLER_IP=10.20.0.2
 144  * PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
 145  * PWD=/home/opnfv
 146  * SHLVL=1
 147  * HOME=/home/opnfv
 148  * NODE_NAME=opnfv-jump-2
 149  * creds=/home/opnfv/functest/conf/openstack.creds
 150  * _=/usr/bin/env
 151
 152 The prepare_env.sh will source the credentials, so once run you should have
 153 access to the following env variables::
 154
 155  root@373f77816eb0:~# env|grep OS_
 156  OS_REGION_NAME=RegionOne
 157  OS_PROJECT_NAME=admin
 158  OS_PASSWORD=admin
 159  OS_AUTH_STRATEGY=keystone
 160  OS_AUTH_URL=http://172.30.10.70:5000/v2.0
 161  OS_USERNAME=admin
 162  OS_TENANT_NAME=admin
 163  OS_ENDPOINT_TYPE=internalURL
 164  OS_NO_CACHE=true
 165
 166
 167 Then you may run the test suite by running::
 168
 169  $ /home/opnfv/repos/functest/docker/run_tests.sh -t <your project>
 170
 171 see `[2]`_ for details.
 172
 173
 174
 175 =========================
 176 How to integrate Functest
 177 =========================
 178
 179 The files of the Functest repository you must modify to integrate Functest are:
 180
 181  * functest/docker/Dockerfile
 182  * functest/docker/common.sh
 183  * functest/docker/requirements.pip
 184  * functest/docker/run_tests.sh
 185  * functest/docker/prepare_env.sh
 186  * functest/config_funtest.yaml
 187
 188
 189 Dockerfile
 190 ==========
 191
 192 This file lists the repositories to be cloned in the Functest container.
 193 The repositories can be internal or external::
 194
 195  RUN git clone https://gerrit.opnfv.org/gerrit/<your porject> ${repos_dir}/<your project>
 196
 197
 198 Common.sh
 199 ==========
 200
 201 This file can be used to declare the branch and commit  variables of your
 202 projects::
 203
 204  <YOUR_PROJECT>_BRANCH=$(cat $config_file | grep -w <your project>_branch | awk 'END {print $NF}')
 205  <YOUR_PROJECT>_COMMIT=$(cat $config_file | grep -w <your project>_commit | awk 'END {print $NF}')
 206
 207  echo "<YOUR_PROJECT>_BRANCH=${<YOUR_PROJECT>_BRANCH}"
 208  echo "<YOUR_PROJECT>_COMMIT=${<YOUR_PROJECT>_COMMIT}"
 209
 210
 211 requirements.pip
 212 ================
 213
 214 This file can be used to preloaded all the needed Python libraries (and avoid
 215 that each project does it)
 216 The current libraries used in Functest container are::
 217
 218  # cat requirements.pip
 219  pyyaml==3.10
 220  gitpython==1.0.1
 221  python-neutronclient==2.6.0
 222  python-novaclient==2.28.1
 223  python-glanceclient==1.1.0
 224  python-cinderclient==1.4.0
 225  python-ceilometerclient==1.5.1
 226  python-keystoneclient==1.6.0
 227  virtualenv==1.11.4
 228  pexpect==4.0
 229  requests==2.8.0
 230  robotframework==2.9.1
 231  robotframework-requests==0.3.8
 232  robotframework-sshlibrary==2.1.1
 233  configObj==5.0.6
 234  Flask==0.10.1
 235  xmltodict==0.9.2
 236  scp==0.10.2
 237  paramiko==1.16.0
 238
 239
 240 prepare_env.sh
 241 ==============
 242
 243 This script can be adapted if you need to set up a specific environment before
 244 running the tests.
 245
 246 run_tests.sh
 247 ============
 248
 249 This script is used to run the tests. You must thus complete the cases with your
 250 own project::
 251
 252  ;;
 253     "promise")
 254         info "Running PROMISE test case..."
 255         # TODO
 256     ;;
 257     "doctor")
 258         info "Running Doctor test..."
 259         python ${FUNCTEST_REPO_DIR}/testcases/features/doctor.py
 260     ;;
 261     "<your project>")
 262         info "Running <your project> test..."
 263         # your script that launchs your tests...
 264     ;;
 265
 266 And do not forget to update also the help line::
 267
 268  -t|--test         run specific set of tests
 269       <test_name>     one or more of the following separated by comma:
 270                          vping_ssh,vping_userdata,odl,rally,tempest,vims,onos,promise,ovno
 271
 272
 273 config_funtest.yaml
 274 ===================
 275
 276 This file is the main configuration file of Functest. You must add the
 277 references to your project::
 278
 279  general:
 280     directories:
 281         dir_repo_<your project>:      /home/opnfv/repos/<your project>
 282      repositories:
 283         # branch and commit ID to which the repos will be reset (HEAD)
 284         <your project>_branch:   master
 285         <your project>_commit:   latest
 286
 287
 288 ====================
 289 Test Dashboard & API
 290 ====================
 291
 292 The OPNFV testing group created a test collection database to collect the test
 293 results from CI.
 294 Any test project running on any lab integrated in CI can push the results to
 295 this database.
 296 This database can be used afterwards to see the evolution of the tests and
 297 compare the results versus the installers, the scenario or the labs.
 298
 299 You can find more information about the dashboard from Testing Dashboard wiki
 300 page `[3]`_.
 301
 302 Overall Architecture
 303 ====================
 304 The Test result management in Brahmaputra can be summarized as follows::
 305
 306   +-------------+    +-------------+    +-------------+
 307   |             |    |             |    |             |
 308   |   Test      |    |   Test      |    |   Test      |
 309   | Project #1  |    | Project #2  |    | Project #N  |
 310   |             |    |             |    |             |
 311   +-------------+    +-------------+    +-------------+
 312            |               |               |
 313            V               V               V
 314        +-----------------------------------------+
 315        |                                         |
 316        |         Test Rest API front end         |
 317        |  http://testresults.opnfv.org/testapi   |
 318        |                                         |
 319        +-----------------------------------------+
 320            A                |
 321            |                V
 322            |     +-------------------------+
 323            |     |                         |
 324            |     |    Test Results DB      |
 325            |     |         Mongo DB        |
 326            |     |                         |
 327            |     +-------------------------+
 328            |
 329            |
 330      +----------------------+
 331      |                      |
 332      |    test Dashboard    |
 333      |                      |
 334      +----------------------+
 335
 336 The Test dashboard URL is: TODO LF
 337 A alternate Test dashboard has been realized to provide a view per installer and
 338 per scenario for Brahmaputra release::
 339
 340   http://testresults.opnfv.org/proto/brahma/
 341
 342 This Dashboard consumes the results retrieved thanks to the Test API.
 343
 344 Test API description
 345 ====================
 346 The Test API is used to declare pods, projects, test cases and test results. An
 347 additional method dashboard has been added to post-process the raw results. The
 348 data model is very basic, 4 objects are created:
 349
 350   * Pods
 351   * Test projects
 352   * Test cases
 353   * Test results
 354
 355 Pods::
 356
 357   {
 358     "id": <ID>,
 359     "details": <URL description of the POD>,
 360     "creation_date": YYYY-MM-DD HH:MM:SS ,
 361     "name": <The POD Name>,
 362     "mode": <metal or virtual>
 363   },
 364
 365 Test project::
 366
 367   {
 368     "id": <ID>,
 369     "name": <Name of the Project>,
 370     "creation_date": "YYYY-MM-DD HH:MM:SS",
 371     "description": <Short description>
 372   },
 373
 374 Test case::
 375
 376   {
 377     "id": <ID>,
 378     "name":<Name of the test case>,
 379     "creation_date": "YYYY-MM-DD HH:MM:SS",
 380     "description": <short description>,
 381     "url":<URL for longer description>
 382   },
 383
 384 Test results::
 385
 386   {
 387     "_id": <ID,
 388     "project_name": <Reference to project>,
 389     "pod_name": <Reference to POD where the test was executed>,
 390     "version": <Scenario on which the test was executed>,
 391     "installer": <Installer Apex or Compass or Fuel or Joid>,
 392     "description": <Short description>,
 393     "creation_date": "YYYY-MM-DD HH:MM:SS",
 394     "case_name": <Reference to the test case>
 395     "details":{
 396        <- the results to be put here ->
 397     }
 398
 399
 400 For Brahmaputra, we got:
 401
 402  * 16 pods
 403  * 18 projects
 404  * 101 test cases
 405
 406 The projects and the test cases have been frozen in December.
 407 But all were not ready for Brahmaputra.
 408
 409
 410
 411 The API can described as follows:
 412
 413 Version:
 414
 415  +--------+--------------------------+-----------------------------------------+
 416  | Method | Path                     | Description                             |
 417  +========+==========================+=========================================+
 418  | GET    | /version                 | Get API version                         |
 419  +--------+--------------------------+-----------------------------------------+
 420
 421
 422 Pods:
 423
 424  +--------+--------------------------+-----------------------------------------+
 425  | Method | Path                     | Description                             |
 426  +========+==========================+=========================================+
 427  | GET    | /pods                    | Get the list of declared Labs (PODs)    |
 428  +--------+--------------------------+-----------------------------------------+
 429  | POST   | /pods                    | Declare a new POD                       |
 430  |        |                          | Content-Type: application/json          |
 431  |        |                          | {                                       |
 432  |        |                          | "name": "pod_foo",                      |
 433  |        |                          | "creation_date": "YYYY-MM-DD HH:MM:SS"  |
 434  |        |                          | }                                       |
 435  +--------+--------------------------+-----------------------------------------+
 436
 437 Projects:
 438
 439  +--------+--------------------------+-----------------------------------------+
 440  | Method | Path                     | Description                             |
 441  +========+==========================+=========================================+
 442  | GET    | /test_projects           | Get the list of test projects           |
 443  +--------+--------------------------+-----------------------------------------+
 444  | GET    |/test_projects/{project}  | Get details on {project}                |
 445  |        |                          |                                         |
 446  +--------+--------------------------+-----------------------------------------+
 447  | POST   | /test_projects           | Add a new test project                  |
 448  |        |                          | Content-Type: application/json          |
 449  |        |                          | {                                       |
 450  |        |                          | "name": "project_foo",                  |
 451  |        |                          | "description": "whatever you want"      |
 452  |        |                          | }                                       |
 453  +--------+--------------------------+-----------------------------------------+
 454  | PUT    | /test_projects/{project} | Update a test project                   |
 455  |        |                          |                                         |
 456  |        |                          | Content-Type: application/json          |
 457  |        |                          | {                                       |
 458  |        |                          | <the field(s) you want to modify>       |
 459  |        |                          | }                                       |
 460  +--------+--------------------------+-----------------------------------------+
 461  | DELETE | /test_projects/{project} | Delete a test project                   |
 462  +--------+--------------------------+-----------------------------------------+
 463
 464
 465 Test cases:
 466
 467  +--------+--------------------------+-----------------------------------------+
 468  | Method | Path                     | Description                             |
 469  +========+==========================+=========================================+
 470  | GET    | /test_projects/{project}/| Get the list of test cases of {project} |
 471  |        | cases                    |                                         |
 472  +--------+--------------------------+-----------------------------------------+
 473  | POST   | /test_projects/{project}/| Add a new test case to {project}        |
 474  |        | cases                    | Content-Type: application/json          |
 475  |        |                          | {                                       |
 476  |        |                          | "name": "case_foo",                     |
 477  |        |                          | "description": "whatever you want"      |
 478  |        |                          | "creation_date": "YYYY-MM-DD HH:MM:SS"  |
 479  |        |                          | "url": "whatever you want"              |
 480  |        |                          | }                                       |
 481  +--------+--------------------------+-----------------------------------------+
 482  | PUT    | /test_projects/{project}?| Modify a test case of {project}         |
 483  |        | case_name={case}         |                                         |
 484  |        |                          | Content-Type: application/json          |
 485  |        |                          | {                                       |
 486  |        |                          | <the field(s) you want to modify>       |
 487  |        |                          | }                                       |
 488  +--------+--------------------------+-----------------------------------------+
 489  | DELETE | /test_projects/{project}/| Delete a test case                      |
 490  |        | case_name={case}         |                                         |
 491  +--------+--------------------------+-----------------------------------------+
 492
 493 Test Results:
 494
 495  +--------+--------------------------+-----------------------------------------+
 496  | Method | Path                     | Description                             |
 497  +========+==========================+=========================================+
 498  | GET    |/results/project={project}| Get the test results of {project}       |
 499  +--------+--------------------------+-----------------------------------------+
 500  | GET    |/results/case={case}      | Get the test results of {case}          |
 501  +--------+--------------------------+-----------------------------------------+
 502  | GET    |/results?pod={pod}        | get the results on pod {pod}            |
 503  +--------+--------------------------+-----------------------------------------+
 504  | GET    |/results?installer={inst} | Get the test results of installer {inst}|
 505  +--------+--------------------------+-----------------------------------------+
 506  | GET    |/results?version={version}| Get the test results of scenario        |
 507  |        |                          | {version}. Initially the version param  |
 508  |        |                          | was reflecting git version, in Functest |
 509  |        |                          | it was decided to move to scenario      |
 510  +--------+--------------------------+-----------------------------------------+
 511  | GET    |/results?project={project}| Get all the results of the test case    |
 512  |        |&case={case}              | {case} of the project {project} with    |
 513  |        |&version={scenario}       | version {scenario} installed by         |
 514  |        |&installer={installer}    | {installer} on POD {pod} stored since   |
 515  |        |&pod={pod}                | {days} days                             |
 516  |        |                          | {project_name} and {case_name} are      |
 517  |        |&period={days}            | mandatory, the other parameters are     |
 518  |        |                          | optional.                               |
 519  +--------+--------------------------+-----------------------------------------+
 520  | POST   | /results                 | Add a new test results                  |
 521  |        |                          | Content-Type: application/json          |
 522  |        |                          | {                                       |
 523  |        |                          | "project_name": "project_foo",          |
 524  |        |                          | "case_name": "case_foo",                |
 525  |        |                          | "pod_name": "pod_foo",                  |
 526  |        |                          | "installer": "installer_foo",           |
 527  |        |                          | "version": "scenario_foo",              |
 528  |        |                          | "details": <your results>               |
 529  |        |                          | }                                       |
 530  +--------+--------------------------+-----------------------------------------+
 531
 532
 533 Dashboard:
 534
 535  +--------+--------------------------+-----------------------------------------+
 536  | Method | Path                     | Description                             |
 537  +========+==========================+=========================================+
 538  | GET    |/dashboard?               | Get all the dashboard ready results of  |
 539  |        |&project={project}        | {case} of the project {project}         |
 540  |        |&case={case}              | version {scenario} installed by         |
 541  |        |&version={scenario}       | {installer} on POD {pod} stored since   |
 542  |        |&installer={installer}    | {days} days                             |
 543  |        |&pod={pod}                |                                         |
 544  |        |&period={days}            | {project_name} and {case_name} are      |
 545  |        |                          | mandatory, the other parameters are     |
 546  |        |                          | optional.                               |
 547  +--------+--------------------------+-----------------------------------------+
 548
 549
 550 The results with dashboard method are post-processed from raw results.
 551 Please note that dashboard results are not stored. Only raw results are stored.
 552
 553
 554 ===============================================
 555 How to push your results into the Test Database
 556 ===============================================
 557
 558 The test database is used to collect test results. By default it is enabled only
 559 in Continuous Integration.
 560 The architecture and associated API is described in `[2]`_.
 561 If you want to push your results from CI, you just have to use the API at the
 562 end of your script.
 563
 564 You can also reuse a python function defined in functest_utils.py::
 565
 566   def push_results_to_db(db_url, case_name, logger, pod_name,version, payload):
 567     """
 568     POST results to the Result target DB
 569     """
 570     url = db_url + "/results"
 571     installer = get_installer_type(logger)
 572     params = {"project_name": "functest", "case_name": case_name,
 573               "pod_name": pod_name, "installer": installer,
 574               "version": version, "details": payload}
 575
 576     headers = {'Content-Type': 'application/json'}
 577     try:
 578         r = requests.post(url, data=json.dumps(params), headers=headers)
 579         if logger:
 580             logger.debug(r)
 581         return True
 582     except Exception, e:
 583         print "Error [push_results_to_db('%s', '%s', '%s', '%s', '%s')]:" \
 584             % (db_url, case_name, pod_name, version, payload), e
 585         return False
 586
 587
 588 ==========
 589 References
 590 ==========
 591
 592 .. _`[1]`: http://artifacts.opnfv.org/functest/docs/configguide/index.html Functest configuration guide URL
 593 .. _`[2]`: http://artifacts.opnfv.org/functest/docs/userguide/index.html functest user guide URL
 594 .. _`[3]`: https://wiki.opnfv.org/opnfv_test_dashboard
 595
 596
 597 OPNFV main site: opnfvmain_.
 598
 599 OPNFV functional test page: opnfvfunctest_.
 600
 601 IRC support chan: #opnfv-testperf
 602
 603 .. _opnfvmain: http://www.opnfv.org
 604 .. _opnfvfunctest: https://wiki.opnfv.org/opnfv_functional_testing
 605 .. _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html
 606 .. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html
 607 .. _`config_test.py` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.py
 608 .. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml
 609