docs/devguide/index.rst

   1 ******************************
   2 OPNFV FUNCTEST developer guide
   3 ******************************
   4
   5 .. toctree::
   6    :numbered:
   7    :maxdepth: 2
   8
   9
  10 ============
  11 Introduction
  12 ============
  13
  14 This document describes how feature projects aiming to run functional tests can
  15 be integrated into FuncTest framework.
  16
  17
  18 ================================
  19 Functest High level architecture
  20 ================================
  21
  22 Functest is project delivering a test container dedicated to OPNFV.
  23 It includes the tools, the scripts and the test scenarios.
  24
  25 Functest can be described as follow::
  26
  27  +----------------------+
  28  |                      |
  29  |   +--------------+   |                  +-------------------+
  30  |   |              |   |    Public        |                   |
  31  |   | Tools        |   +------------------+      OPNFV        |
  32  |   | Scripts      |   |                  | System Under Test |
  33  |   | Scenarios    |   +------------------+                   |
  34  |   |              |   |    Management    |                   |
  35  |   +--------------+   |                  +-------------------+
  36  |                      |
  37  |    Functest Docker   |
  38  |                      |
  39  +----------------------+
  40
  41 Functest deals with internal and external test cases.
  42 The Internal test cases in Brahmaputra are:
  43
  44  * vPing
  45  * ODL
  46  * Tempest
  47  * vIMS
  48  * Rally
  49
  50 The external tescases are:
  51
  52  * Promise
  53  * Doctor
  54  * Onos
  55  * BGPVPN
  56
  57 see `[2]`_ for details.
  58
  59 Functest can also be considered as a framework that may include external OPNFV
  60 projects.
  61 This framework will ease the integration of the feature test suite to the CI.
  62
  63 ==================
  64 How Functest works
  65 ==================
  66
  67 The installation and the launch of the Functest docker image is described in
  68 `[1]`_.
  69
  70 The Functest docker directories are::
  71
  72  home
  73     |
  74     `-- opnfv
  75         |-- functest
  76         |   |-- conf
  77         |   |-- data
  78         |   `-- results
  79         `-- repos
  80             |-- bgpvpn
  81             |-- doctor
  82             |-- functest
  83             |-- odl_integration
  84             |-- onos
  85             |-- ovno
  86             |-- promise
  87             |-- rally
  88             |-- releng
  89             `-- vims-test
  90
  91 ::
  92
  93  +-----------+-------------------+---------------------------------------------+
  94  | Directory | Subdirectory      | Comments                                    |
  95  +-----------+-------------------+---------------------------------------------+
  96  |           | <project>/conf    | All the useful configuration file(s) for    |
  97  |           |                   | <project> the openstack creds are put there |
  98  |           |                   | for CI                                      |
  99  |           |                   | It is recommended to push it there when     |
 100  |           |                   | passing the credentials to container through|
 101  |           |                   | the -v option                               |
 102  |           +-------------------+---------------------------------------------+
 103  |    opnfv  | <project>/data    | Usefull data, images for <projects>         |
 104  |           |                   | By default we put a cirros image:           |
 105  |           |                   | cirros-0.3.4-x86_64-disk.img                |
 106  |           |                   | This image can be used by any projects      |
 107  |           +-------------------+---------------------------------------------+
 108  |           | <project>/results | Local result directory of project <project> |
 109  +-----------+-------------------+---------------------------------------------+
 110  |           | bgpvpn            |                                             |
 111  |           +-------------------+                                             +
 112  | repos     | doctor            |                                             |
 113  |           +-------------------+                                             +
 114  |           | functest          |                                             |
 115  |           +-------------------+                                             +
 116  |           | odl_integration   |                                             |
 117  |           +-------------------+    Clone of the useful repositories         +
 118  |           | onos              |     These repositories may include:         |
 119  |           +-------------------+                - tooling                    +
 120  |           | promise           |                - scenario                   |
 121  |           +-------------------+                - scripts                    +
 122  |           | rally             |                                             |
 123  |           +-------------------+                                             +
 124  |           | releng            |                                             |
 125  |           +-------------------+                                             +
 126  |           | vims-test         |                                             |
 127  |           +-------------------+                                             +
 128  |           | <your project>    |                                             |
 129  +-----------+-------------------+---------------------------------------------+
 130
 131 Before running the test suite, you must prepare the environement by running::
 132
 133  $ /home/opnfv/repos/functest/docker/prepare_env.sh
 134
 135 By running prepare_env.sh, you build the test environement required by the tests
 136 including the retrieval and sourcing of OpenStack credentials.
 137 This is an example of the env variables we have in the docker container:
 138
 139  * HOSTNAME=373f77816eb0
 140  * INSTALLER_TYPE=fuel
 141  * repos_dir=/home/opnfv/repos
 142  * INSTALLER_IP=10.20.0.2
 143  * PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
 144  * PWD=/home/opnfv
 145  * SHLVL=1
 146  * HOME=/home/opnfv
 147  * NODE_NAME=opnfv-jump-2
 148  * creds=/home/opnfv/functest/conf/openstack.creds
 149  * _=/usr/bin/env
 150
 151 The prepare_env.sh will source the credentials, so once run you should have
 152 access to the following env variables::
 153
 154  root@373f77816eb0:~# env|grep OS_
 155  OS_REGION_NAME=RegionOne
 156  OS_PROJECT_NAME=admin
 157  OS_PASSWORD=admin
 158  OS_AUTH_STRATEGY=keystone
 159  OS_AUTH_URL=http://172.30.10.70:5000/v2.0
 160  OS_USERNAME=admin
 161  OS_TENANT_NAME=admin
 162  OS_ENDPOINT_TYPE=internalURL
 163  OS_NO_CACHE=true
 164
 165
 166 Then you may run the test suite by running::
 167
 168  $ /home/opnfv/repos/functest/docker/run_tests.sh -t <your project>
 169
 170 see `[2]`_ for details.
 171
 172
 173
 174 =========================
 175 How to integrate Functest
 176 =========================
 177
 178 The files of the Functest repository you must modify to integrate Functest are:
 179
 180  * functest/docker/Dockerfile
 181  * functest/docker/common.sh
 182  * functest/docker/requirements.pip
 183  * functest/docker/run_tests.sh
 184  * functest/docker/prepare_env.sh
 185  * functest/config_funtest.yaml
 186
 187
 188 Dockerfile
 189 ==========
 190
 191 This file lists the repositories to be cloned in the Functest container.
 192 The repositories can be internal or external::
 193
 194  RUN git clone https://gerrit.opnfv.org/gerrit/<your porject> ${repos_dir}/<your project>
 195
 196
 197 Common.sh
 198 ==========
 199
 200 This file can be used to declare the branch and commit  variables of your
 201 projects::
 202
 203  <YOUR_PROJECT>_BRANCH=$(cat $config_file | grep -w <your project>_branch | awk 'END {print $NF}')
 204  <YOUR_PROJECT>_COMMIT=$(cat $config_file | grep -w <your project>_commit | awk 'END {print $NF}')
 205
 206  echo "<YOUR_PROJECT>_BRANCH=${<YOUR_PROJECT>_BRANCH}"
 207  echo "<YOUR_PROJECT>_COMMIT=${<YOUR_PROJECT>_COMMIT}"
 208
 209
 210 requirements.pip
 211 ================
 212
 213 This file can be used to preloaded all the needed Python libraries (and avoid
 214 that each project does it)
 215 The current libraries used in Functest container are::
 216
 217  # cat requirements.pip
 218  pyyaml==3.10
 219  gitpython==1.0.1
 220  python-neutronclient==2.6.0
 221  python-novaclient==2.28.1
 222  python-glanceclient==1.1.0
 223  python-cinderclient==1.4.0
 224  python-ceilometerclient==1.5.1
 225  python-keystoneclient==1.6.0
 226  virtualenv==1.11.4
 227  pexpect==4.0
 228  requests==2.8.0
 229  robotframework==2.9.1
 230  robotframework-requests==0.3.8
 231  robotframework-sshlibrary==2.1.1
 232  configObj==5.0.6
 233  Flask==0.10.1
 234  xmltodict==0.9.2
 235  scp==0.10.2
 236  paramiko==1.16.0
 237
 238
 239 prepare_env.sh
 240 ==============
 241
 242 This script can be adapted if you need to set up a specific environment before
 243 running the tests.
 244
 245 run_tests.sh
 246 ============
 247
 248 This script is used to run the tests. You must thus complete the cases with your
 249 own project::
 250
 251  ;;
 252     "promise")
 253         info "Running PROMISE test case..."
 254         # TODO
 255     ;;
 256     "doctor")
 257         info "Running Doctor test..."
 258         python ${FUNCTEST_REPO_DIR}/testcases/features/doctor.py
 259     ;;
 260     "<your project>")
 261         info "Running <your project> test..."
 262         # your script that launchs your tests...
 263     ;;
 264
 265 And do not forget to update also the help line::
 266
 267  -t|--test         run specific set of tests
 268       <test_name>     one or more of the following: vping,odl,rally,tempest,vims,onos, promise. Separated by comma.
 269
 270
 271 config_funtest.yaml
 272 ===================
 273
 274 This file is the main configuration file of Functest. You must add the
 275 references to your project::
 276
 277  general:
 278     directories:
 279         dir_repo_<your project>:      /home/opnfv/repos/<your project>
 280      repositories:
 281         # branch and commit ID to which the repos will be reset (HEAD)
 282         <your project>_branch:   master
 283         <your project>_commit:   latest
 284
 285
 286 ====================
 287 Test Dashboard & API
 288 ====================
 289
 290 The OPNFV testing group created a test collection database to collect the test
 291 results from CI.
 292 Any test project running on any lab integrated in CI can push the results to
 293 this database.
 294 This database can be used afterwards to see the evolution of the tests and
 295 compare the results versus the installers, the scenario or the labs.
 296
 297 You can find more information about the dashboard from Testing Dashboard wiki
 298 page `[3]`_.
 299
 300
 301 Overall Architecture
 302 --------------------
 303
 304 The Test result management in Brahmaputra can be summarized as follows::
 305
 306   +-------------+    +-------------+    +-------------+
 307   |             |    |             |    |             |
 308   |   Test      |    |   Test      |    |   Test      |
 309   | Project #1  |    | Project #2  |    | Project #N  |
 310   |             |    |             |    |             |
 311   +-------------+    +-------------+    +-------------+
 312            |               |               |
 313            V               V               V
 314        +-----------------------------------------+
 315        |                                         |
 316        |         Test Rest API front end         |
 317        |  http://testresults.opnfv.org/testapi   |
 318        |                                         |
 319        +-----------------------------------------+
 320            A                |
 321            |                V
 322            |     +-------------------------+
 323            |     |                         |
 324            |     |    Test Results DB      |
 325            |     |         Mongo DB        |
 326            |     |                         |
 327            |     +-------------------------+
 328            |
 329            |
 330      +----------------------+
 331      |                      |
 332      |    test Dashboard    |
 333      |                      |
 334      +----------------------+
 335
 336 The Test dashboard URL is: TODO LF
 337 A alternate Test dashboard has been realized to provide a view per installer and
 338 per scenario for Brahmaputra release::
 339
 340   http://testresults.opnfv.org/proto/brahma/
 341
 342 This Dashboard consumes the results retrieved thanks to the Test API.
 343
 344 Test API description
 345 --------------------
 346
 347 The Test API is used to declare pods, projects, test cases and test results. An
 348 additional method dashboard has been added to post-process the raw results. The
 349 data model is very basic, 4 objects are created:
 350
 351   * Pods
 352   * Test projects
 353   * Test cases
 354   * Test results
 355
 356 Pods::
 357
 358   {
 359     "id": <ID>,
 360     "details": <URL description of the POD>,
 361     "creation_date": YYYY-MM-DD HH:MM:SS ,
 362     "name": <The POD Name>,
 363     "mode": <metal or virtual>
 364   },
 365
 366 Test project::
 367
 368   {
 369     "id": <ID>,
 370     "name": <Name of the Project>,
 371     "creation_date": "YYYY-MM-DD HH:MM:SS",
 372     "description": <Short description>
 373   },
 374
 375 Test case::
 376
 377   {
 378     "id": <ID>,
 379     "name":<Name of the test case>,
 380     "creation_date": "YYYY-MM-DD HH:MM:SS",
 381     "description": <short description>,
 382     "url":<URL for longer description>
 383   },
 384
 385 Test results::
 386
 387   {
 388     "_id": <ID,
 389     "project_name": <Reference to project>,
 390     "pod_name": <Reference to POD where the test was executed>,
 391     "version": <Scenario on which the test was executed>,
 392     "installer": <Installer Apex or Compass or Fuel or Joid>,
 393     "description": <Short description>,
 394     "creation_date": "YYYY-MM-DD HH:MM:SS",
 395     "case_name": <Reference to the test case>
 396     "details":{
 397        <- the results to be put here ->
 398     }
 399
 400
 401 For Brahmaputra, we got:
 402
 403  * 16 pods
 404  * 18 projects
 405  * 101 test cases
 406
 407 The projects and the test cases have been frozen in December.
 408 But all were not ready for Brahmaputra.
 409
 410
 411
 412 The API can described as follows:
 413
 414 Version:
 415
 416  +--------+--------------------------+-----------------------------------------+
 417  | Method | Path                     | Description                             |
 418  +========+==========================+=========================================+
 419  | GET    | /version                 | Get API version                         |
 420  +--------+--------------------------+-----------------------------------------+
 421
 422
 423 Pods:
 424
 425  +--------+--------------------------+-----------------------------------------+
 426  | Method | Path                     | Description                             |
 427  +========+==========================+=========================================+
 428  | GET    | /pods                    | Get the list of declared Labs (PODs)    |
 429  +--------+--------------------------+-----------------------------------------+
 430  | POST   | /pods                    | Declare a new POD                       |
 431  |        |                          | Content-Type: application/json          |
 432  |        |                          | {                                       |
 433  |        |                          |   "name": "pod_foo",                    |
 434  |        |                          |   "creation_date": "YYYY-MM-DD HH:MM:SS"|
 435  |        |                          | }                                       |
 436  +--------+--------------------------+-----------------------------------------+
 437
 438 Projects:
 439
 440  +--------+--------------------------+-----------------------------------------+
 441  | Method | Path                     | Description                             |
 442  +========+==========================+=========================================+
 443  | GET    | /test_projects           | Get the list of test projects           |
 444  +--------+--------------------------+-----------------------------------------+
 445  | GET    |/test_projects/{project}  | Get details on {project}                |
 446  |        |                          |                                         |
 447  +--------+--------------------------+-----------------------------------------+
 448  | POST   | /test_projects           | Add a new test project                  |
 449  |        |                          | Content-Type: application/json          |
 450  |        |                          | {                                       |
 451  |        |                          |    "name": "project_foo",               |
 452  |        |                          |    "description": "whatever you want"   |
 453  |        |                          | }                                       |
 454  +--------+--------------------------+-----------------------------------------+
 455  | PUT    | /test_projects/{project} | Update a test project                   |
 456  |        |                          |                                         |
 457  |        |                          | Content-Type: application/json          |
 458  |        |                          | {                                       |
 459  |        |                          |    <the field(s) you want to modify>    |
 460  |        |                          | }                                       |
 461  +--------+--------------------------+-----------------------------------------+
 462  | DELETE | /test_projects/{project} | Delete a test project                   |
 463  +--------+--------------------------+-----------------------------------------+
 464
 465
 466 Test cases:
 467
 468  +--------+--------------------------+-----------------------------------------+
 469  | Method | Path                     | Description                             |
 470  +========+==========================+=========================================+
 471  | GET    | /test_projects/{project}/| Get the list of test cases of {project} |
 472  |        | cases                    |                                         |
 473  +--------+--------------------------+-----------------------------------------+
 474  | POST   | /test_projects/{project}/| Add a new test case to {project}        |
 475  |        | cases                    | Content-Type: application/json          |
 476  |        |                          | {                                       |
 477  |        |                          |   "name": "case_foo",                   |
 478  |        |                          |   "description": "whatever you want"    |
 479  |        |                          |   "creation_date": "YYYY-MM-DD HH:MM:SS"|
 480  |        |                          |   "url": "whatever you want"            |
 481  |        |                          | }                                       |
 482  +--------+--------------------------+-----------------------------------------+
 483  | PUT    | /test_projects/{project}?| Modify a test case of {project}         |
 484  |        | case_name={case}         |                                         |
 485  |        |                          | Content-Type: application/json          |
 486  |        |                          | {                                       |
 487  |        |                          |    <the field(s) you want to modify>    |
 488  |        |                          | }                                       |
 489  +--------+--------------------------+-----------------------------------------+
 490  | DELETE | /test_projects/{project}/| Delete a test case                      |
 491  |        | case_name={case}         |                                         |
 492  +--------+--------------------------+-----------------------------------------+
 493
 494 Test Results:
 495
 496  +--------+--------------------------+-----------------------------------------+
 497  | Method | Path                     | Description                             |
 498  +========+==========================+=========================================+
 499  | GET    |/results/project={project}| Get the test results of {project}       |
 500  +--------+--------------------------+-----------------------------------------+
 501  | GET    |/results/case={case}      | Get the test results of {case}          |
 502  +--------+--------------------------+-----------------------------------------+
 503  | GET    |/results?pod={pod}        | get the results on pod {pod}            |
 504  +--------+--------------------------+-----------------------------------------+
 505  | GET    |/results?installer={inst} | Get the test results of installer {inst}|
 506  +--------+--------------------------+-----------------------------------------+
 507  | GET    |/results?version={version}| Get the test results of scenario        |
 508  |        |                          | {version}. Initially the version param  |
 509  |        |                          | was reflecting git version, in Functest |
 510  |        |                          | it was decided to move to scenario      |
 511  +--------+--------------------------+-----------------------------------------+
 512  | GET    |/results?project={project}| Get all the results of the test case    |
 513  |        |&case={case}              | {case} of the project {project} with    |
 514  |        |&version={scenario}       | version {scenario} installed by         |
 515  |        |&installer={installer}    | {installer} on POD {pod} stored since   |
 516  |        |&pod={pod}                | {days} days                             |
 517  |        |                          | {project_name} and {case_name} are      |
 518  |        |&period={days}            | mandatory, the other parameters are     |
 519  |        |                          | optional.                               |
 520  +--------+--------------------------+-----------------------------------------+
 521  | POST   | /results                 | Add a new test results                  |
 522  |        |                          | Content-Type: application/json          |
 523  |        |                          | {                                       |
 524  |        |                          |    "project_name": "project_foo",       |
 525  |        |                          |    "case_name": "case_foo",             |
 526  |        |                          |    "pod_name": "pod_foo",               |
 527  |        |                          |    "installer": "installer_foo",        |
 528  |        |                          |    "version": "scenario_foo",           |
 529  |        |                          |    "details": <your results>            |
 530  |        |                          | }                                       |
 531  +--------+--------------------------+-----------------------------------------+
 532
 533
 534 Dashboard:
 535
 536  +--------+--------------------------+-----------------------------------------+
 537  | Method | Path                     | Description                             |
 538  +========+==========================+=========================================+
 539  | GET    |/dashboard?               | Get all the dashboard ready results of  |
 540  |        |&project={project}        | {case} of the project {project}         |
 541  |        |&case={case}              | version {scenario} installed by         |
 542  |        |&version={scenario}       | {installer} on POD {pod} stored since   |
 543  |        |&installer={installer}    | {days} days                             |
 544  |        |&pod={pod}                |                                         |
 545  |        |&period={days}            | {project_name} and {case_name} are      |
 546  |        |                          | mandatory, the other parameters are     |
 547  |        |                          | optional.                               |
 548  +--------+--------------------------+-----------------------------------------+
 549
 550
 551 The results with dashboard method are post-processed from raw results.
 552 Please note that dashboard results are not stored. Only raw results are stored.
 553
 554
 555 ===============================================
 556 How to push your results into the Test Database
 557 ===============================================
 558
 559 The test database is used to collect test results. By default it is enabled only
 560 in Continuous Integration.
 561 The architecture and associated API is described in `[2]`_.
 562 If you want to push your results from CI, you just have to use the API at the
 563 end of your script.
 564
 565 You can also reuse a python function defined in functest_utils.py::
 566
 567   def push_results_to_db(db_url, case_name, logger, pod_name,version, payload):
 568     """
 569     POST results to the Result target DB
 570     """
 571     url = db_url + "/results"
 572     installer = get_installer_type(logger)
 573     params = {"project_name": "functest", "case_name": case_name,
 574               "pod_name": pod_name, "installer": installer,
 575               "version": version, "details": payload}
 576
 577     headers = {'Content-Type': 'application/json'}
 578     try:
 579         r = requests.post(url, data=json.dumps(params), headers=headers)
 580         if logger:
 581             logger.debug(r)
 582         return True
 583     except Exception, e:
 584         print "Error [push_results_to_db('%s', '%s', '%s', '%s', '%s')]:" \
 585             % (db_url, case_name, pod_name, version, payload), e
 586         return False
 587
 588 ::
 589
 590 ==========
 591 References
 592 ==========
 593
 594 .. _`[1]`: http://artifacts.opnfv.org/functest/docs/configguide/index.html Functest configuration guide URL
 595 .. _`[2]`: http://artifacts.opnfv.org/functest/docs/userguide/index.html functest user guide URL
 596 .. _`[3]`: https://wiki.opnfv.org/opnfv_test_dashboard
 597
 598
 599 OPNFV main site: opnfvmain_.
 600
 601 OPNFV functional test page: opnfvfunctest_.
 602
 603 IRC support chan: #opnfv-testperf
 604
 605 .. _opnfvmain: http://www.opnfv.org
 606 .. _opnfvfunctest: https://wiki.opnfv.org/opnfv_functional_testing
 607 .. _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html
 608 .. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html
 609 .. _`config_test.py` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.py
 610 .. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml
 611