docs/devguide/index.rst

   1 ******************************
   2 OPNFV FUNCTEST developer guide
   3 ******************************
   4
   5 .. toctree::
   6    :numbered:
   7    :maxdepth: 2
   8
   9
  10 ============
  11 Introduction
  12 ============
  13
  14 This document describes how feature projects aiming to run functional tests can
  15 be integrated into FuncTest framework.
  16
  17
  18 ================================
  19 Functest High level architecture
  20 ================================
  21
  22 Functest is project delivering a test container dedicated to OPNFV.
  23 It includes the tools, the scripts and the test scenarios.
  24
  25 Functest can be described as follow::
  26
  27  +----------------------+
  28  |                      |
  29  |   +--------------+   |                  +-------------------+
  30  |   |              |   |    Public        |                   |
  31  |   | Tools        |   +------------------+      OPNFV        |
  32  |   | Scripts      |   |                  | System Under Test |
  33  |   | Scenarios    |   +------------------+                   |
  34  |   |              |   |    Management    |                   |
  35  |   +--------------+   |                  +-------------------+
  36  |                      |
  37  |    Functest Docker   |
  38  |                      |
  39  +----------------------+
  40
  41 Functest deals with internal and external test cases.
  42 The Internal test cases in Brahmaputra are:
  43
  44  * vPing_SSH
  45  * vPing_userdata
  46  * ODL
  47  * Tempest
  48  * vIMS
  49  * Rally
  50
  51 The external tescases are:
  52
  53  * Promise
  54  * Doctor
  55  * Onos
  56  * BGPVPN
  57
  58 see `[2]`_ for details.
  59
  60 Functest can also be considered as a framework that may include external OPNFV
  61 projects.
  62 This framework will ease the integration of the feature test suite to the CI.
  63
  64 ==================
  65 How Functest works
  66 ==================
  67
  68 The installation and the launch of the Functest docker image is described in
  69 `[1]`_.
  70
  71 The Functest docker directories are::
  72
  73  home
  74     |
  75     `-- opnfv
  76         |-- functest
  77         |   |-- conf
  78         |   |-- data
  79         |   `-- results
  80         `-- repos
  81             |-- bgpvpn
  82             |-- doctor
  83             |-- functest
  84             |-- odl_integration
  85             |-- onos
  86             |-- ovno
  87             |-- promise
  88             |-- rally
  89             |-- releng
  90             `-- vims-test
  91
  92 ::
  93
  94  +-----------+-------------------+---------------------------------------------+
  95  | Directory | Subdirectory      | Comments                                    |
  96  +-----------+-------------------+---------------------------------------------+
  97  |           | <project>/conf    | All the useful configuration file(s) for    |
  98  |           |                   | <project> the openstack creds are put there |
  99  |           |                   | for CI                                      |
 100  |           |                   | It is recommended to push it there when     |
 101  |           |                   | passing the credentials to container through|
 102  |           |                   | the -v option                               |
 103  |           +-------------------+---------------------------------------------+
 104  |    opnfv  | <project>/data    | Usefull data, images for <projects>         |
 105  |           |                   | By default we put a cirros image:           |
 106  |           |                   | cirros-0.3.4-x86_64-disk.img                |
 107  |           |                   | This image can be used by any projects      |
 108  |           +-------------------+---------------------------------------------+
 109  |           | <project>/results | Local result directory of project <project> |
 110  +-----------+-------------------+---------------------------------------------+
 111  |           | bgpvpn            |                                             |
 112  |           +-------------------+                                             +
 113  | repos     | doctor            |                                             |
 114  |           +-------------------+                                             +
 115  |           | functest          |                                             |
 116  |           +-------------------+                                             +
 117  |           | odl_integration   |                                             |
 118  |           +-------------------+    Clone of the useful repositories         +
 119  |           | onos              |     These repositories may include:         |
 120  |           +-------------------+                - tooling                    +
 121  |           | promise           |                - scenario                   |
 122  |           +-------------------+                - scripts                    +
 123  |           | rally             |                                             |
 124  |           +-------------------+                                             +
 125  |           | releng            |                                             |
 126  |           +-------------------+                                             +
 127  |           | vims-test         |                                             |
 128  |           +-------------------+                                             +
 129  |           | <your project>    |                                             |
 130  +-----------+-------------------+---------------------------------------------+
 131
 132 Before running the test suite, you must prepare the environement by running::
 133
 134  $ /home/opnfv/repos/functest/docker/prepare_env.sh
 135
 136 By running prepare_env.sh, you build the test environement required by the tests
 137 including the retrieval and sourcing of OpenStack credentials.
 138 This is an example of the env variables we have in the docker container:
 139
 140  * HOSTNAME=373f77816eb0
 141  * INSTALLER_TYPE=fuel
 142  * repos_dir=/home/opnfv/repos
 143  * INSTALLER_IP=10.20.0.2
 144  * PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
 145  * PWD=/home/opnfv
 146  * SHLVL=1
 147  * HOME=/home/opnfv
 148  * NODE_NAME=opnfv-jump-2
 149  * creds=/home/opnfv/functest/conf/openstack.creds
 150  * _=/usr/bin/env
 151
 152 The prepare_env.sh will source the credentials, so once run you should have
 153 access to the following env variables::
 154
 155  root@373f77816eb0:~# env|grep OS_
 156  OS_REGION_NAME=RegionOne
 157  OS_PROJECT_NAME=admin
 158  OS_PASSWORD=admin
 159  OS_AUTH_STRATEGY=keystone
 160  OS_AUTH_URL=http://172.30.10.70:5000/v2.0
 161  OS_USERNAME=admin
 162  OS_TENANT_NAME=admin
 163  OS_ENDPOINT_TYPE=internalURL
 164  OS_NO_CACHE=true
 165
 166
 167 Then you may run the test suite by running::
 168
 169  $ /home/opnfv/repos/functest/docker/run_tests.sh -t <your project>
 170
 171 see `[2]`_ for details.
 172
 173
 174
 175 =========================
 176 How to integrate Functest
 177 =========================
 178
 179 The files of the Functest repository you must modify to integrate Functest are:
 180
 181  * functest/docker/Dockerfile
 182  * functest/docker/common.sh
 183  * functest/docker/requirements.pip
 184  * functest/docker/run_tests.sh
 185  * functest/docker/prepare_env.sh
 186  * functest/config_funtest.yaml
 187
 188
 189 Dockerfile
 190 ==========
 191
 192 This file lists the repositories to be cloned in the Functest container.
 193 The repositories can be internal or external::
 194
 195  RUN git clone https://gerrit.opnfv.org/gerrit/<your porject> ${repos_dir}/<your project>
 196
 197
 198 Common.sh
 199 ==========
 200
 201 This file can be used to declare the branch and commit  variables of your
 202 projects::
 203
 204  <YOUR_PROJECT>_BRANCH=$(cat $config_file | grep -w <your project>_branch | awk 'END {print $NF}')
 205  <YOUR_PROJECT>_COMMIT=$(cat $config_file | grep -w <your project>_commit | awk 'END {print $NF}')
 206
 207  echo "<YOUR_PROJECT>_BRANCH=${<YOUR_PROJECT>_BRANCH}"
 208  echo "<YOUR_PROJECT>_COMMIT=${<YOUR_PROJECT>_COMMIT}"
 209
 210
 211 requirements.pip
 212 ================
 213
 214 This file can be used to preloaded all the needed Python libraries (and avoid
 215 that each project does it)
 216 The current libraries used in Functest container are::
 217
 218  # cat requirements.pip
 219  pyyaml==3.10
 220  gitpython==1.0.1
 221  python-neutronclient==2.6.0
 222  python-novaclient==2.28.1
 223  python-glanceclient==1.1.0
 224  python-cinderclient==1.4.0
 225  python-ceilometerclient==1.5.1
 226  python-keystoneclient==1.6.0
 227  virtualenv==1.11.4
 228  pexpect==4.0
 229  requests==2.8.0
 230  robotframework==2.9.1
 231  robotframework-requests==0.3.8
 232  robotframework-sshlibrary==2.1.1
 233  configObj==5.0.6
 234  Flask==0.10.1
 235  xmltodict==0.9.2
 236  scp==0.10.2
 237  paramiko==1.16.0
 238
 239
 240 prepare_env.sh
 241 ==============
 242
 243 This script can be adapted if you need to set up a specific environment before
 244 running the tests.
 245
 246 run_tests.sh
 247 ============
 248
 249 This script is used to run the tests. You must thus complete the cases with your
 250 own project::
 251
 252  ;;
 253     "promise")
 254         info "Running PROMISE test case..."
 255         # TODO
 256     ;;
 257     "doctor")
 258         info "Running Doctor test..."
 259         python ${FUNCTEST_REPO_DIR}/testcases/features/doctor.py
 260     ;;
 261     "<your project>")
 262         info "Running <your project> test..."
 263         # your script that launchs your tests...
 264     ;;
 265
 266 And do not forget to update also the help line::
 267
 268  -t|--test         run specific set of tests
 269       <test_name>     one or more of the following separated by comma:
 270                          vping_ssh,vping_userdata,odl,rally,tempest,vims,onos,promise,ovno
 271
 272
 273 config_funtest.yaml
 274 ===================
 275
 276 This file is the main configuration file of Functest. You must add the
 277 references to your project::
 278
 279  general:
 280     directories:
 281         dir_repo_<your project>:      /home/opnfv/repos/<your project>
 282      repositories:
 283         # branch and commit ID to which the repos will be reset (HEAD)
 284         <your project>_branch:   master
 285         <your project>_commit:   latest
 286
 287
 288 ====================
 289 Test Dashboard & API
 290 ====================
 291
 292 The OPNFV testing group created a test collection database to collect the test
 293 results from CI.
 294 Any test project running on any lab integrated in CI can push the results to
 295 this database.
 296 This database can be used afterwards to see the evolution of the tests and
 297 compare the results versus the installers, the scenario or the labs.
 298
 299 You can find more information about the dashboard from Testing Dashboard wiki
 300 page `[3]`_.
 301
 302
 303 Overall Architecture
 304 --------------------
 305
 306 The Test result management in Brahmaputra can be summarized as follows::
 307
 308   +-------------+    +-------------+    +-------------+
 309   |             |    |             |    |             |
 310   |   Test      |    |   Test      |    |   Test      |
 311   | Project #1  |    | Project #2  |    | Project #N  |
 312   |             |    |             |    |             |
 313   +-------------+    +-------------+    +-------------+
 314            |               |               |
 315            V               V               V
 316        +-----------------------------------------+
 317        |                                         |
 318        |         Test Rest API front end         |
 319        |  http://testresults.opnfv.org/testapi   |
 320        |                                         |
 321        +-----------------------------------------+
 322            A                |
 323            |                V
 324            |     +-------------------------+
 325            |     |                         |
 326            |     |    Test Results DB      |
 327            |     |         Mongo DB        |
 328            |     |                         |
 329            |     +-------------------------+
 330            |
 331            |
 332      +----------------------+
 333      |                      |
 334      |    test Dashboard    |
 335      |                      |
 336      +----------------------+
 337
 338 The Test dashboard URL is: TODO LF
 339 A alternate Test dashboard has been realized to provide a view per installer and
 340 per scenario for Brahmaputra release::
 341
 342   http://testresults.opnfv.org/proto/brahma/
 343
 344 This Dashboard consumes the results retrieved thanks to the Test API.
 345
 346 Test API description
 347 --------------------
 348
 349 The Test API is used to declare pods, projects, test cases and test results. An
 350 additional method dashboard has been added to post-process the raw results. The
 351 data model is very basic, 4 objects are created:
 352
 353   * Pods
 354   * Test projects
 355   * Test cases
 356   * Test results
 357
 358 Pods::
 359
 360   {
 361     "id": <ID>,
 362     "details": <URL description of the POD>,
 363     "creation_date": YYYY-MM-DD HH:MM:SS ,
 364     "name": <The POD Name>,
 365     "mode": <metal or virtual>
 366   },
 367
 368 Test project::
 369
 370   {
 371     "id": <ID>,
 372     "name": <Name of the Project>,
 373     "creation_date": "YYYY-MM-DD HH:MM:SS",
 374     "description": <Short description>
 375   },
 376
 377 Test case::
 378
 379   {
 380     "id": <ID>,
 381     "name":<Name of the test case>,
 382     "creation_date": "YYYY-MM-DD HH:MM:SS",
 383     "description": <short description>,
 384     "url":<URL for longer description>
 385   },
 386
 387 Test results::
 388
 389   {
 390     "_id": <ID,
 391     "project_name": <Reference to project>,
 392     "pod_name": <Reference to POD where the test was executed>,
 393     "version": <Scenario on which the test was executed>,
 394     "installer": <Installer Apex or Compass or Fuel or Joid>,
 395     "description": <Short description>,
 396     "creation_date": "YYYY-MM-DD HH:MM:SS",
 397     "case_name": <Reference to the test case>
 398     "details":{
 399        <- the results to be put here ->
 400     }
 401
 402
 403 For Brahmaputra, we got:
 404
 405  * 16 pods
 406  * 18 projects
 407  * 101 test cases
 408
 409 The projects and the test cases have been frozen in December.
 410 But all were not ready for Brahmaputra.
 411
 412
 413
 414 The API can described as follows:
 415
 416 Version:
 417
 418  +--------+--------------------------+-----------------------------------------+
 419  | Method | Path                     | Description                             |
 420  +========+==========================+=========================================+
 421  | GET    | /version                 | Get API version                         |
 422  +--------+--------------------------+-----------------------------------------+
 423
 424
 425 Pods:
 426
 427  +--------+--------------------------+-----------------------------------------+
 428  | Method | Path                     | Description                             |
 429  +========+==========================+=========================================+
 430  | GET    | /pods                    | Get the list of declared Labs (PODs)    |
 431  +--------+--------------------------+-----------------------------------------+
 432  | POST   | /pods                    | Declare a new POD                       |
 433  |        |                          | Content-Type: application/json          |
 434  |        |                          | {                                       |
 435  |        |                          |   "name": "pod_foo",                    |
 436  |        |                          |   "creation_date": "YYYY-MM-DD HH:MM:SS"|
 437  |        |                          | }                                       |
 438  +--------+--------------------------+-----------------------------------------+
 439
 440 Projects:
 441
 442  +--------+--------------------------+-----------------------------------------+
 443  | Method | Path                     | Description                             |
 444  +========+==========================+=========================================+
 445  | GET    | /test_projects           | Get the list of test projects           |
 446  +--------+--------------------------+-----------------------------------------+
 447  | GET    |/test_projects/{project}  | Get details on {project}                |
 448  |        |                          |                                         |
 449  +--------+--------------------------+-----------------------------------------+
 450  | POST   | /test_projects           | Add a new test project                  |
 451  |        |                          | Content-Type: application/json          |
 452  |        |                          | {                                       |
 453  |        |                          |    "name": "project_foo",               |
 454  |        |                          |    "description": "whatever you want"   |
 455  |        |                          | }                                       |
 456  +--------+--------------------------+-----------------------------------------+
 457  | PUT    | /test_projects/{project} | Update a test project                   |
 458  |        |                          |                                         |
 459  |        |                          | Content-Type: application/json          |
 460  |        |                          | {                                       |
 461  |        |                          |    <the field(s) you want to modify>    |
 462  |        |                          | }                                       |
 463  +--------+--------------------------+-----------------------------------------+
 464  | DELETE | /test_projects/{project} | Delete a test project                   |
 465  +--------+--------------------------+-----------------------------------------+
 466
 467
 468 Test cases:
 469
 470  +--------+--------------------------+-----------------------------------------+
 471  | Method | Path                     | Description                             |
 472  +========+==========================+=========================================+
 473  | GET    | /test_projects/{project}/| Get the list of test cases of {project} |
 474  |        | cases                    |                                         |
 475  +--------+--------------------------+-----------------------------------------+
 476  | POST   | /test_projects/{project}/| Add a new test case to {project}        |
 477  |        | cases                    | Content-Type: application/json          |
 478  |        |                          | {                                       |
 479  |        |                          |   "name": "case_foo",                   |
 480  |        |                          |   "description": "whatever you want"    |
 481  |        |                          |   "creation_date": "YYYY-MM-DD HH:MM:SS"|
 482  |        |                          |   "url": "whatever you want"            |
 483  |        |                          | }                                       |
 484  +--------+--------------------------+-----------------------------------------+
 485  | PUT    | /test_projects/{project}?| Modify a test case of {project}         |
 486  |        | case_name={case}         |                                         |
 487  |        |                          | Content-Type: application/json          |
 488  |        |                          | {                                       |
 489  |        |                          |    <the field(s) you want to modify>    |
 490  |        |                          | }                                       |
 491  +--------+--------------------------+-----------------------------------------+
 492  | DELETE | /test_projects/{project}/| Delete a test case                      |
 493  |        | case_name={case}         |                                         |
 494  +--------+--------------------------+-----------------------------------------+
 495
 496 Test Results:
 497
 498  +--------+--------------------------+-----------------------------------------+
 499  | Method | Path                     | Description                             |
 500  +========+==========================+=========================================+
 501  | GET    |/results/project={project}| Get the test results of {project}       |
 502  +--------+--------------------------+-----------------------------------------+
 503  | GET    |/results/case={case}      | Get the test results of {case}          |
 504  +--------+--------------------------+-----------------------------------------+
 505  | GET    |/results?pod={pod}        | get the results on pod {pod}            |
 506  +--------+--------------------------+-----------------------------------------+
 507  | GET    |/results?installer={inst} | Get the test results of installer {inst}|
 508  +--------+--------------------------+-----------------------------------------+
 509  | GET    |/results?version={version}| Get the test results of scenario        |
 510  |        |                          | {version}. Initially the version param  |
 511  |        |                          | was reflecting git version, in Functest |
 512  |        |                          | it was decided to move to scenario      |
 513  +--------+--------------------------+-----------------------------------------+
 514  | GET    |/results?project={project}| Get all the results of the test case    |
 515  |        |&case={case}              | {case} of the project {project} with    |
 516  |        |&version={scenario}       | version {scenario} installed by         |
 517  |        |&installer={installer}    | {installer} on POD {pod} stored since   |
 518  |        |&pod={pod}                | {days} days                             |
 519  |        |                          | {project_name} and {case_name} are      |
 520  |        |&period={days}            | mandatory, the other parameters are     |
 521  |        |                          | optional.                               |
 522  +--------+--------------------------+-----------------------------------------+
 523  | POST   | /results                 | Add a new test results                  |
 524  |        |                          | Content-Type: application/json          |
 525  |        |                          | {                                       |
 526  |        |                          |    "project_name": "project_foo",       |
 527  |        |                          |    "case_name": "case_foo",             |
 528  |        |                          |    "pod_name": "pod_foo",               |
 529  |        |                          |    "installer": "installer_foo",        |
 530  |        |                          |    "version": "scenario_foo",           |
 531  |        |                          |    "details": <your results>            |
 532  |        |                          | }                                       |
 533  +--------+--------------------------+-----------------------------------------+
 534
 535
 536 Dashboard:
 537
 538  +--------+--------------------------+-----------------------------------------+
 539  | Method | Path                     | Description                             |
 540  +========+==========================+=========================================+
 541  | GET    |/dashboard?               | Get all the dashboard ready results of  |
 542  |        |&project={project}        | {case} of the project {project}         |
 543  |        |&case={case}              | version {scenario} installed by         |
 544  |        |&version={scenario}       | {installer} on POD {pod} stored since   |
 545  |        |&installer={installer}    | {days} days                             |
 546  |        |&pod={pod}                |                                         |
 547  |        |&period={days}            | {project_name} and {case_name} are      |
 548  |        |                          | mandatory, the other parameters are     |
 549  |        |                          | optional.                               |
 550  +--------+--------------------------+-----------------------------------------+
 551
 552
 553 The results with dashboard method are post-processed from raw results.
 554 Please note that dashboard results are not stored. Only raw results are stored.
 555
 556
 557 ===============================================
 558 How to push your results into the Test Database
 559 ===============================================
 560
 561 The test database is used to collect test results. By default it is enabled only
 562 in Continuous Integration.
 563 The architecture and associated API is described in `[2]`_.
 564 If you want to push your results from CI, you just have to use the API at the
 565 end of your script.
 566
 567 You can also reuse a python function defined in functest_utils.py::
 568
 569   def push_results_to_db(db_url, case_name, logger, pod_name,version, payload):
 570     """
 571     POST results to the Result target DB
 572     """
 573     url = db_url + "/results"
 574     installer = get_installer_type(logger)
 575     params = {"project_name": "functest", "case_name": case_name,
 576               "pod_name": pod_name, "installer": installer,
 577               "version": version, "details": payload}
 578
 579     headers = {'Content-Type': 'application/json'}
 580     try:
 581         r = requests.post(url, data=json.dumps(params), headers=headers)
 582         if logger:
 583             logger.debug(r)
 584         return True
 585     except Exception, e:
 586         print "Error [push_results_to_db('%s', '%s', '%s', '%s', '%s')]:" \
 587             % (db_url, case_name, pod_name, version, payload), e
 588         return False
 589
 590 ::
 591
 592 ==========
 593 References
 594 ==========
 595
 596 .. _`[1]`: http://artifacts.opnfv.org/functest/docs/configguide/index.html Functest configuration guide URL
 597 .. _`[2]`: http://artifacts.opnfv.org/functest/docs/userguide/index.html functest user guide URL
 598 .. _`[3]`: https://wiki.opnfv.org/opnfv_test_dashboard
 599
 600
 601 OPNFV main site: opnfvmain_.
 602
 603 OPNFV functional test page: opnfvfunctest_.
 604
 605 IRC support chan: #opnfv-testperf
 606
 607 .. _opnfvmain: http://www.opnfv.org
 608 .. _opnfvfunctest: https://wiki.opnfv.org/opnfv_functional_testing
 609 .. _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html
 610 .. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html
 611 .. _`config_test.py` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.py
 612 .. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml
 613