testcases/VIM/OpenStack/OpenStack.md

   1 # Rally tests for OpenStack
   2
   3 Original Rally testsuites can be found here: https://github.com/stackforge/rally
   4
   5 ---
   6 ## Intro
   7 In order to perform functional and performance testing, we use Rally (see https://wiki.openstack.org/wiki/Rally for details).
   8 Rally must be installed as jenkins user on the jumphost machine of the OPNFV solution.
   9
  10 ## Installation & Configuration
  11
  12 ### Rally
  13
  14 * Log on jumphost machine as jenkins user
  15 * Create the file existing.json, adapt it to your OpenStack (until agreement on default passwords)
  16 ```bash
  17 {
  18     "type": "ExistingCloud",
  19     "auth_url": "http://example.net:5000/v2.0/",
  20     "region_name": "RegionOne",
  21     "endpoint_type": "public",
  22     "admin": {
  23         "username": "admin",
  24         "password": "myadminpass",
  25         "tenant_name": "demo"
  26     },
  27     "https_insecure": False,
  28     "https_cacert": "",
  29 }
  30 ```
  31 * Install Rally (ref https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html)
  32
  33 ```bash
  34 git clone https://git.openstack.org/stackforge/rally
  35 ./rally/install_rally.sh -v
  36 rally deployment create --file=existing.json --name=existing
  37 ```
  38 * you can check the available OpenStack services
  39 ```bash
  40 # rally deployment check
  41 keystone endpoints are valid and following service are available:
  42 +-------------+-----------+------------+
  43 | Services  | Type        | Status     |
  44 +-----------+-------------+------------+
  45 | cinder    | volume      | Available  |
  46 | cinderv2  | volumev2    | Available  |
  47 | glance    | image       | Available  |
  48 | keystone  | identity    | Available  |
  49 | neutron   | network     | Available  |
  50 | nova      | compute     | Available  |
  51 | nova_ec2  | compute_ec2 | Available  |
  52 | novav3    | computev3   | Available  |
  53 +-----------+-------------+------------+
  54 ```
  55 * You can start Rally scenario manually, follow https://rally.readthedocs.org/en/latest/tutorial/step_1_setting_up_env_and_running_benchmark_from_samples.html
  56 ```bash
  57 # rally task start ./samples/tasks/scenarios/nova/my-boot-and-delete.json
  58 --------------------------------------------------------------------------------
  59  Preparing input task
  60 --------------------------------------------------------------------------------
  61
  62 Input task is:
  63 {
  64     "NovaServers.boot_and_delete_server": [
  65         {
  66             "args": {
  67                 "flavor": {
  68                     "name": "m1.small"
  69                 },
  70                 "image": {
  71                     "name": "^ubuntu-14.10-64b"
  72                 },
  73                 "force_delete": false
  74             },
  75             "runner": {
  76                 "type": "constant",
  77                 "times": 10,
  78                 "concurrency": 2
  79             },
  80             "context": {
  81                 "users": {
  82                     "tenants": 3,
  83                     "users_per_tenant": 2
  84                 }
  85             }
  86         }
  87     ]
  88 }
  89
  90 --------------------------------------------------------------------------------
  91  Task  f42c8aed-00a6-4715-9951-945b4fb97c32: started
  92 --------------------------------------------------------------------------------
  93
  94 Benchmarking... This can take a while...
  95
  96 To track task status use:
  97
  98         rally task status
  99         or
 100         rally task detailed
 101
 102 --------------------------------------------------------------------------------
 103 Task f42c8aed-00a6-4715-9951-945b4fb97c32: finished
 104 --------------------------------------------------------------------------------
 105
 106 test scenario NovaServers.boot_and_delete_server
 107 args position 0
 108 args values:
 109 OrderedDict([(u'runner', OrderedDict([(u'type', u'constant'), (u'concurrency', 2), (u'times', 10)])), (u'args', OrderedDict([(u'force_delete', False), (u'flavor', OrderedDict([(u'name', u'm1.small')])), (u'image', OrderedDict([(u'name', u'^ubuntu-14.10-64b')]))])), (u'context', OrderedDict([(u'users', OrderedDict([(u'project_domain', u'default'), (u'users_per_tenant', 2), (u'tenants', 3), (u'resource_management_workers', 30), (u'user_domain', u'default')]))]))])
 110 +--------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
 111 | action             | min (sec) | avg (sec) | max (sec) | 90 percentile | 95 percentile | success | count |
 112 +--------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
 113 | nova.boot_server   | 4.675     | 5.554     | 6.357     | 6.289         | 6.323         | 100.0%  | 10    |
 114 | nova.delete_server | 2.365     | 3.301     | 4.728     | 4.553         | 4.64          | 100.0%  | 10    |
 115 | total              | 7.303     | 8.857     | 10.789    | 10.543        | 10.666        | 100.0%  | 10    |
 116 +--------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
 117 Load duration: 45.7972288132
 118 Full duration: 58.912060976
 119
 120 HINTS:
 121 * To plot HTML graphics with this data, run:
 122         rally task report f42c8aed-00a6-4715-9951-945b4fb97c32 --out output.html
 123
 124 * To get raw JSON output of task results, run:
 125         rally task results f42c8aed-00a6-4715-9951-945b4fb97c32
 126
 127 Using task: f42c8aed-00a6-4715-9951-945b4fb97c32
 128
 129 ```
 130 * For Tempest, you can run the test manually by following the instructions https://www.mirantis.com/blog/rally-openstack-tempest-testing-made-simpler
 131
 132 ```bash
 133 root@rally:~/rally# rally verify start
 134 [...]
 135 tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest
 136     test_attach_volumes_with_nonexistent_volume_id[compute,gate,id-f5e56b0a-5d02-43c1-a2a7-c9b792c2e3f6,negative]FAIL
 137     test_create_volume_with_invalid_size[gate,id-1ed83a8a-682d-4dfb-a30e-ee63ffd6c049,negative]OK  0.02
 138     test_create_volume_with_nonexistent_snapshot_id[gate,id-0c36f6ae-4604-4017-b0a9-34fdc63096f9,negative]OK  0.04
 139     test_create_volume_with_nonexistent_source_volid[gate,id-47c73e08-4be8-45bb-bfdf-0c4e79b88344,negative]OK  0.05
 140     test_create_volume_with_nonexistent_volume_type[gate,id-10254ed8-3849-454e-862e-3ab8e6aa01d2,negative]OK  0.02
 141     test_create_volume_with_out_passing_size[gate,id-9387686f-334f-4d31-a439-33494b9e2683,negative]OK  0.02
 142     test_create_volume_with_size_negative[gate,id-8b472729-9eba-446e-a83b-916bdb34bef7,negative]OK  0.02
 143 [...]
 144 Ran 933 tests in 1020.200s
 145
 146 FAILED (failures=186)
 147 Test set 'full' has been finished with error. Check log for details
 148
 149 ```
 150
 151 It is possible to get a better view on the result
 152 ```bash
 153 # rally verify list
 154 +--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------------+----------+
 155 | UUID                                 | Deployment UUID                      | Set name | Tests | Failures | Created at                 | Duration       | Status   |
 156 +--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------------+----------+
 157 | b1de3608-dbee-40e7-84c4-1c756ca0347c | e7d70ddf-9be0-4681-9456-aa8dce515e0e | None     | 0     | 0        | 2015-03-11 08:48:04.416793 | 0:00:00.102275 | running  |
 158 | ff0d9285-184f-47d5-9474-7475135ae8cf | e7d70ddf-9be0-4681-9456-aa8dce515e0e | full     | 933   | 186      | 2015-03-11 09:57:01.836611 | 0:18:08.360204 | finished |
 159 | fec2fd0a-a4ef-4064-a292-95e9da68025c | e7d70ddf-9be0-4681-9456-aa8dce515e0e | full     | 933   | 186      | 2015-03-12 09:46:40.818691 | 0:17:02.316443 | finished |
 160 +--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------------+----------+
 161
 162 rally verify show fec2fd0a-a4ef-4064-a292-95e9da68025c
 163 Total results of verification:
 164
 165 +--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+
 166 | UUID                                 | Deployment UUID                      | Set name | Tests | Failures | Created at                 | Status   |
 167 +--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+
 168 | fec2fd0a-a4ef-4064-a292-95e9da68025c | e7d70ddf-9be0-4681-9456-aa8dce515e0e | full     | 933   | 186      | 2015-03-12 09:46:40.818691 | finished |
 169 +--------------------------------------+--------------------------------------+----------+-------+----------+----------------------------+----------+
 170
 171 Tests:
 172
 173 +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+--------+
 174 | name                                                                                                                                                                                                      | time      | status |
 175 +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+--------+
 176 | tearDownClass (tempest.api.image.v1.test_images.CreateRegisterImagesTest)                                                                                                                                 | 0.0       | FAIL   |
 177 | tearDownClass (tempest.api.image.v1.test_images.UpdateImageMetaTest)                                                                                                                                      | 0.0       | FAIL   |
 178 [...]
 179 | tempest.cli.simple_read_only.volume.test_cinder.SimpleReadOnlyCinderClientTest.test_cinder_quota_show[id-18166673-ffa8-4df3-b60c-6375532288bc]                                                            | 1.309555  | OK     |
 180 | tempest.cli.simple_read_only.volume.test_cinder.SimpleReadOnlyCinderClientTest.test_cinder_rate_limits[id-b2c66ed9-ca96-4dc4-94cc-8083e664e516]                                                           | 1.277704  | OK     |
 181 | tempest.cli.simple_read_only.volume.test_cinder.SimpleReadOnlyCinderClientTest.test_cinder_region_list[id-95a2850c-35b4-4159-bb93-51647a5ad232]                                                           | 1.105877  | FAIL   |
 182 | tempest.cli.simple_read_only.volume.test_cinder.SimpleReadOnlyCinderClientTest.test_cinder_retries_list[id-6d97fcd2-5dd1-429d-af70-030c949d86cd]                                                          | 1.306407  | OK     |
 183 | tempest.cli.simple_read_only.volume.test_cinder.SimpleReadOnlyCinderClientTest.test_cinder_service_list[id-301b5ae1-9591-4e9f-999c-d525a9bdf822]                                                          | 1.24909   | OK     |
 184 | tempest.cli.simple_read_only.volume.test_cinder.SimpleReadOnlyCinderClientTest.test_cinder_snapshot_list[id-7a19955b-807c-481a-a2ee-9d76733eac28]                                                         | 1.270242  | OK     |
 185 [...]
 186 | tempest.thirdparty.boto.test_s3_ec2_images.S3ImagesTest                                                                                                                                                   | 0.0       | SKIP   |
 187 | tempest.thirdparty.boto.test_s3_objects.S3BucketsTest.test_create_get_delete_object[id-4eea567a-b46a-405b-a475-6097e1faebde]                                                                              | 0.239222  | FAIL   |
 188 +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+--------+
 189
 190 ```
 191 Rally includes a reporting tool
 192 https://rally.readthedocs.org/en/latest/tutorial/step_1_setting_up_env_and_running_benchmark_from_samples.html
 193
 194 ## Test description
 195
 196 ### Rally
 197
 198 By default, the different Rally Scenarios are:
 199 ```bash
 200
 201 ls samples/tasks/scenarios/
 202 authenticate  cinder     dummy   heat      mistral  nova    README.rst  sahara                                 vm
 203 ceilometer    designate  glance  keystone  neutron  quotas  requests    tempest-do-not-run-against-production  zaqar
 204
 205 ```
 206
 207 tempest tests can be retrieved at https://github.com/openstack/tempest
 208
 209 tests have been grouped and are available in https://git.opnfv.org/cgit/functest/tree/testcases/VIM/OpenStack/CI/suites
 210
 211
 212 ## Automation
 213
 214 For automation, 2 job-templates have been created in https://git.opnfv.org/cgit/releng/tree/jjb/functest/functest.yml
 215
 216 * functest-vim_bench-test: this template runs automatically a python script that runs the different rally scenario (except Tempest)
 217 * functest-vim_tempest-test: this template runs the rally command rally verify start
 218