Update scenario img_file in tempest.conf
[functest.git] / README.md
1 # Functest
2
3 Network virtualization has dramatically modified our architectures which asks
4 for more automation and powerful testing tools like Functest, a collection of
5 state-of-the-art virtual infrastructure test suites, including automatic VNF
6 testing (cf.
7 [[1]](https://www.linuxfoundation.org/press-release/2019/05/opnfv-hunter-delivers-test-tools-ci-cd-framework-to-enable-common-nfvi-for-verifying-vnfs/)).
8
9 In context of OPNFV, Functest verifies any kind of OpenStack and Kubernetes
10 deployments including production environments. It conforms to upstream rules
11 and integrates smoothly lots of the test cases available in the opensource
12 market. It includes about 3000+ functional tests and 3 hours upstream API and
13 dataplane benchmarks. It’s completed by Virtual Network Function deployments
14 and testing (vIMS, vRouter and vEPC) to ensure that the platforms meet Network
15 Functions Virtualization requirements. Raspberry PI is also supported to verify
16 datacenters as the lowest cost (50 euros hardware and software included).
17
18 | Functest releases | OpenStack releases |
19 |-------------------|--------------------|
20 | Hunter                  | Rocky              |
21 | Iruya             | Stein              |
22 | Jerma             | Train              |
23 | Kali              | Ussuri             |
24 | Leguer            | Victoria           |
25 | **Master**          | **next Wallaby**   |
26
27 ## Prepare your environment
28
29 cat env
30 ```
31 DEPLOY_SCENARIO=XXX  # if not os-nosdn-nofeature-noha scenario
32 NAMESERVER=XXX  # if not 8.8.8.8
33 EXTERNAL_NETWORK=XXX  # if not first network with router:external=True
34 DASHBOARD_URL=XXX  # else tempest_horizon will be skipped
35 NEW_USER_ROLE=XXX  # if not member
36 SDN_CONTROLLER_IP=XXX  # if odl scenario
37 VOLUME_DEVICE_NAME=XXX  # if not vdb
38 FLAVOR_EXTRA_SPECS=hw:mem_page_size:large  # if fdio scenarios
39 ```
40
41 cat openstack.creds
42 ```
43 export OS_AUTH_URL=XXX
44 export OS_USER_DOMAIN_NAME=XXX
45 export OS_PROJECT_DOMAIN_NAME=XXX
46 export OS_USERNAME=XXX
47 export OS_PROJECT_NAME=XXX
48 export OS_PASSWORD=XXX
49 export OS_IDENTITY_API_VERSION=3
50 export OS_REGION_NAME=XXX
51 ```
52
53 mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh | bash -s -- images && ls -1 images/*
54 ```
55 images/cirros-0.5.1-aarch64-disk.img
56 images/cirros-0.5.1-x86_64-disk.img
57 images/cloudify-docker-manager-community-19.01.24.tar
58 images/Fedora-Cloud-Base-30-1.2.x86_64.qcow2
59 images/shaker-image-1.3.0+stretch.qcow2
60 images/ubuntu-14.04-server-cloudimg-amd64-disk1.img
61 images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img
62 images/ubuntu-16.04-server-cloudimg-amd64-disk1.img
63 images/vyos-1.1.8-amd64.qcow2
64 ```
65
66 ## Run healthcheck suite
67
68 ```bash
69 sudo docker run --env-file env \
70     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
71     -v $(pwd)/images:/home/opnfv/functest/images \
72     opnfv/functest-healthcheck
73 ```
74
75 ```
76 +--------------------------+------------------+---------------------+------------------+----------------+
77 |        TEST CASE         |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
78 +--------------------------+------------------+---------------------+------------------+----------------+
79 |     connection_check     |     functest     |     healthcheck     |      00:03       |      PASS      |
80 |      tenantnetwork1      |     functest     |     healthcheck     |      00:05       |      PASS      |
81 |      tenantnetwork2      |     functest     |     healthcheck     |      00:06       |      PASS      |
82 |         vmready1         |     functest     |     healthcheck     |      00:06       |      PASS      |
83 |         vmready2         |     functest     |     healthcheck     |      00:08       |      PASS      |
84 |        singlevm1         |     functest     |     healthcheck     |      00:32       |      PASS      |
85 |        singlevm2         |     functest     |     healthcheck     |      00:37       |      PASS      |
86 |        vping_ssh         |     functest     |     healthcheck     |      00:46       |      PASS      |
87 |      vping_userdata      |     functest     |     healthcheck     |      00:39       |      PASS      |
88 |       cinder_test        |     functest     |     healthcheck     |      01:05       |      PASS      |
89 |      tempest_smoke       |     functest     |     healthcheck     |      05:39       |      PASS      |
90 |     tempest_horizon      |     functest     |     healthcheck     |      01:05       |      PASS      |
91 |           odl            |     functest     |     healthcheck     |      00:00       |      SKIP      |
92 +--------------------------+------------------+---------------------+------------------+----------------+
93 ```
94
95 ## Run smoke suite
96
97 ```bash
98 sudo docker run --env-file env \
99     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
100     -v $(pwd)/images:/home/opnfv/functest/images \
101     opnfv/functest-smoke
102 ```
103
104 ```
105 +---------------------------+------------------+---------------+------------------+----------------+
106 |         TEST CASE         |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
107 +---------------------------+------------------+---------------+------------------+----------------+
108 |      tempest_neutron      |     functest     |     smoke     |      15:30       |      PASS      |
109 |       tempest_cinder      |     functest     |     smoke     |      02:01       |      PASS      |
110 |      tempest_keystone     |     functest     |     smoke     |      01:17       |      PASS      |
111 |        tempest_heat       |     functest     |     smoke     |      22:14       |      PASS      |
112 |     tempest_telemetry     |     functest     |     smoke     |      00:00       |      SKIP      |
113 |        rally_sanity       |     functest     |     smoke     |      17:24       |      PASS      |
114 |      refstack_compute     |     functest     |     smoke     |      07:03       |      PASS      |
115 |      refstack_object      |     functest     |     smoke     |      02:09       |      PASS      |
116 |     refstack_platform     |     functest     |     smoke     |      07:31       |      PASS      |
117 |        tempest_full       |     functest     |     smoke     |      41:52       |      PASS      |
118 |      tempest_scenario     |     functest     |     smoke     |      08:42       |      PASS      |
119 |        tempest_slow       |     functest     |     smoke     |      43:42       |      PASS      |
120 |       patrole_admin       |     functest     |     smoke     |      21:06       |      PASS      |
121 |       patrole_member      |     functest     |     smoke     |      21:23       |      PASS      |
122 |       patrole_reader      |     functest     |     smoke     |      21:56       |      PASS      |
123 |      tempest_barbican     |     functest     |     smoke     |      02:30       |      PASS      |
124 |      tempest_octavia      |     functest     |     smoke     |      00:00       |      SKIP      |
125 |       tempest_cyborg      |     functest     |     smoke     |      00:00       |      SKIP      |
126 +---------------------------+------------------+---------------+------------------+----------------+
127 ```
128
129 ## Run smoke CNTT suite
130
131 ```bash
132 sudo docker run --env-file env \
133     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
134     -v $(pwd)/images:/home/opnfv/functest/images \
135     opnfv/functest-smoke-cntt
136 ```
137
138 ```
139 +-------------------------------+------------------+---------------+------------------+----------------+
140 |           TEST CASE           |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
141 +-------------------------------+------------------+---------------+------------------+----------------+
142 |      tempest_neutron_cntt     |     functest     |     smoke     |      11:35       |      PASS      |
143 |      tempest_cinder_cntt      |     functest     |     smoke     |      01:58       |      PASS      |
144 |     tempest_keystone_cntt     |     functest     |     smoke     |      01:13       |      PASS      |
145 |       tempest_heat_cntt       |     functest     |     smoke     |      22:32       |      PASS      |
146 |       rally_sanity_cntt       |     functest     |     smoke     |      17:16       |      PASS      |
147 |       tempest_full_cntt       |     functest     |     smoke     |      41:13       |      PASS      |
148 |     tempest_scenario_cntt     |     functest     |     smoke     |      08:57       |      PASS      |
149 |       tempest_slow_cntt       |     functest     |     smoke     |      35:58       |      PASS      |
150 +-------------------------------+------------------+---------------+------------------+----------------+
151 ```
152
153 ## Run benchmarking suite
154
155 ```bash
156 sudo docker run --env-file env \
157     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
158     -v $(pwd)/images:/home/opnfv/functest/images \
159     opnfv/functest-benchmarking
160 ```
161
162 ```
163 +--------------------+------------------+----------------------+------------------+----------------+
164 |     TEST CASE      |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
165 +--------------------+------------------+----------------------+------------------+----------------+
166 |     rally_full     |     functest     |     benchmarking     |      93:03       |      PASS      |
167 |     rally_jobs     |     functest     |     benchmarking     |      27:05       |      PASS      |
168 |        vmtp        |     functest     |     benchmarking     |      17:56       |      PASS      |
169 |       shaker       |     functest     |     benchmarking     |      24:02       |      PASS      |
170 +--------------------+------------------+----------------------+------------------+----------------+
171 ```
172
173 ## Run benchmarking CNTT suite
174
175 ```bash
176 sudo docker run --env-file env \
177     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
178     -v $(pwd)/images:/home/opnfv/functest/images \
179     opnfv/functest-benchmarking-cntt
180 ```
181
182 ```
183 +-------------------------+------------------+----------------------+------------------+----------------+
184 |        TEST CASE        |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
185 +-------------------------+------------------+----------------------+------------------+----------------+
186 |     rally_full_cntt     |     functest     |     benchmarking     |      89:52       |      PASS      |
187 |     rally_jobs_cntt     |     functest     |     benchmarking     |      19:39       |      PASS      |
188 |           vmtp          |     functest     |     benchmarking     |      16:59       |      PASS      |
189 |          shaker         |     functest     |     benchmarking     |      23:43       |      PASS      |
190 +-------------------------+------------------+----------------------+------------------+----------------+
191 ```
192
193 ## Run vnf suite
194
195 ```bash
196 sudo docker run --env-file env \
197     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
198     -v $(pwd)/images:/home/opnfv/functest/images \
199     opnfv/functest-vnf
200 ```
201
202 ```
203 +----------------------+------------------+--------------+------------------+----------------+
204 |      TEST CASE       |     PROJECT      |     TIER     |     DURATION     |     RESULT     |
205 +----------------------+------------------+--------------+------------------+----------------+
206 |       cloudify       |     functest     |     vnf      |      05:08       |      PASS      |
207 |     cloudify_ims     |     functest     |     vnf      |      24:46       |      PASS      |
208 |       heat_ims       |     functest     |     vnf      |      33:12       |      PASS      |
209 |     vyos_vrouter     |     functest     |     vnf      |      15:53       |      PASS      |
210 |       juju_epc       |     functest     |     vnf      |      27:52       |      PASS      |
211 +----------------------+------------------+--------------+------------------+----------------+
212 ```