Publish Functest Virtual Machines
[functest.git] / README.md
1 # Functest
2
3 Network virtualization has dramatically modified our architectures which asks
4 for more automation and powerful testing tools like Functest, a collection of
5 state-of-the-art virtual infrastructure test suites, including automatic VNF
6 testing (cf.
7 [[1]](https://www.linuxfoundation.org/press-release/2019/05/opnfv-hunter-delivers-test-tools-ci-cd-framework-to-enable-common-nfvi-for-verifying-vnfs/)).
8
9 In context of OPNFV, Functest verifies any kind of OpenStack and Kubernetes
10 deployments including production environments. It conforms to upstream rules
11 and integrates smoothly lots of the test cases available in the opensource
12 market. It includes about 3000+ functional tests and 3 hours upstream API and
13 dataplane benchmarks. It’s completed by Virtual Network Function deployments
14 and testing (vIMS, vRouter and vEPC) to ensure that the platforms meet Network
15 Functions Virtualization requirements. Raspberry PI is also supported to verify
16 datacenters as the lowest cost (50 euros hardware and software included).
17
18 | Functest releases | OpenStack releases |
19 |-------------------|--------------------|
20 | **Jerma**         | **Train**          |
21 | Kali              | Ussuri             |
22 | Leguer            | Victoria           |
23 | Wallaby           | Wallaby            |
24 | Master            | next Xena          |
25
26 ## Prepare your environment
27
28 cat env
29 ```
30 DEPLOY_SCENARIO=XXX  # if not os-nosdn-nofeature-noha scenario
31 NAMESERVER=XXX  # if not 8.8.8.8
32 EXTERNAL_NETWORK=XXX  # if not first network with router:external=True
33 DASHBOARD_URL=XXX  # else tempest_horizon will be skipped
34 NEW_USER_ROLE=XXX  # if not member
35 SDN_CONTROLLER_IP=XXX  # if odl scenario
36 VOLUME_DEVICE_NAME=XXX  # if not vdb
37 FLAVOR_EXTRA_SPECS=hw:mem_page_size:large  # if fdio scenarios
38 ```
39
40 cat openstack.creds
41 ```
42 export OS_AUTH_URL=XXX
43 export OS_USER_DOMAIN_NAME=XXX
44 export OS_PROJECT_DOMAIN_NAME=XXX
45 export OS_USERNAME=XXX
46 export OS_PROJECT_NAME=XXX
47 export OS_PASSWORD=XXX
48 export OS_IDENTITY_API_VERSION=3
49 export OS_REGION_NAME=XXX
50 ```
51
52 mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh?h=stable/jerma | bash -s -- images && ls -1 images/*
53 ```
54 images/cirros-0.4.0-aarch64-disk.img
55 images/cirros-0.4.0-x86_64-disk.img
56 images/cloudify-docker-manager-community-19.01.24.tar
57 images/Fedora-Cloud-Base-30-1.2.x86_64.qcow2
58 images/shaker-image-1.3.0+stretch.qcow2
59 images/ubuntu-14.04-server-cloudimg-amd64-disk1.img
60 images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img
61 images/ubuntu-16.04-server-cloudimg-amd64-disk1.img
62 images/vyos-1.1.8-amd64.qcow2
63 ```
64
65 ## Run healthcheck suite
66
67 ```bash
68 sudo docker run --env-file env \
69     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
70     -v $(pwd)/images:/home/opnfv/functest/images \
71     opnfv/functest-healthcheck:jerma
72 ```
73
74 ```
75 +--------------------------+------------------+---------------------+------------------+----------------+
76 |        TEST CASE         |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
77 +--------------------------+------------------+---------------------+------------------+----------------+
78 |     connection_check     |     functest     |     healthcheck     |      00:02       |      PASS      |
79 |      tenantnetwork1      |     functest     |     healthcheck     |      00:04       |      PASS      |
80 |      tenantnetwork2      |     functest     |     healthcheck     |      00:05       |      PASS      |
81 |         vmready1         |     functest     |     healthcheck     |      00:05       |      PASS      |
82 |         vmready2         |     functest     |     healthcheck     |      00:08       |      PASS      |
83 |        singlevm1         |     functest     |     healthcheck     |      00:35       |      PASS      |
84 |        singlevm2         |     functest     |     healthcheck     |      00:34       |      PASS      |
85 |        vping_ssh         |     functest     |     healthcheck     |      00:44       |      PASS      |
86 |      vping_userdata      |     functest     |     healthcheck     |      00:30       |      PASS      |
87 |       cinder_test        |     functest     |     healthcheck     |      01:10       |      PASS      |
88 |      tempest_smoke       |     functest     |     healthcheck     |      11:18       |      PASS      |
89 |     tempest_horizon      |     functest     |     healthcheck     |      01:15       |      PASS      |
90 |           odl            |     functest     |     healthcheck     |      00:00       |      SKIP      |
91 +--------------------------+------------------+---------------------+------------------+----------------+
92 ```
93
94 ## Run smoke suite
95
96 ```bash
97 sudo docker run --env-file env \
98     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
99     -v $(pwd)/images:/home/opnfv/functest/images \
100     opnfv/functest-smoke:jerma
101 ```
102
103 ```
104 +---------------------------+------------------+---------------+------------------+----------------+
105 |         TEST CASE         |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
106 +---------------------------+------------------+---------------+------------------+----------------+
107 |      tempest_neutron      |     functest     |     smoke     |      14:49       |      PASS      |
108 |       tempest_cinder      |     functest     |     smoke     |      01:40       |      PASS      |
109 |      tempest_keystone     |     functest     |     smoke     |      01:12       |      PASS      |
110 |        tempest_heat       |     functest     |     smoke     |      21:53       |      PASS      |
111 |        rally_sanity       |     functest     |     smoke     |      17:10       |      PASS      |
112 |      refstack_compute     |     functest     |     smoke     |      07:53       |      PASS      |
113 |      refstack_object      |     functest     |     smoke     |      01:59       |      PASS      |
114 |     refstack_platform     |     functest     |     smoke     |      07:52       |      PASS      |
115 |        tempest_full       |     functest     |     smoke     |      34:39       |      PASS      |
116 |      tempest_scenario     |     functest     |     smoke     |      08:30       |      PASS      |
117 |        tempest_slow       |     functest     |     smoke     |      44:40       |      PASS      |
118 |          patrole          |     functest     |     smoke     |      02:43       |      PASS      |
119 |      tempest_barbican     |     functest     |     smoke     |      02:26       |      PASS      |
120 |      tempest_octavia      |     functest     |     smoke     |      00:00       |      SKIP      |
121 +---------------------------+------------------+---------------+------------------+----------------+
122 ```
123
124 ## Run smoke CNTT suite
125
126 ```bash
127 sudo docker run --env-file env \
128     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
129     -v $(pwd)/images:/home/opnfv/functest/images \
130     opnfv/functest-smoke-cntt:jerma
131 ```
132
133 ```
134 +-------------------------------+------------------+---------------+------------------+----------------+
135 |           TEST CASE           |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
136 +-------------------------------+------------------+---------------+------------------+----------------+
137 |      tempest_neutron_cntt     |     functest     |     smoke     |      12:21       |      PASS      |
138 |      tempest_cinder_cntt      |     functest     |     smoke     |      01:31       |      PASS      |
139 |     tempest_keystone_cntt     |     functest     |     smoke     |      01:08       |      PASS      |
140 |       tempest_heat_cntt       |     functest     |     smoke     |      21:47       |      PASS      |
141 |       rally_sanity_cntt       |     functest     |     smoke     |      16:09       |      PASS      |
142 |       tempest_full_cntt       |     functest     |     smoke     |      39:11       |      PASS      |
143 |     tempest_scenario_cntt     |     functest     |     smoke     |      08:32       |      PASS      |
144 |       tempest_slow_cntt       |     functest     |     smoke     |      32:22       |      PASS      |
145 +-------------------------------+------------------+---------------+------------------+----------------+
146 ```
147
148 ## Run benchmarking suite
149
150 ```bash
151 sudo docker run --env-file env \
152     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
153     -v $(pwd)/images:/home/opnfv/functest/images \
154     opnfv/functest-benchmarking:jerma
155 ```
156
157 ```
158 +--------------------+------------------+----------------------+------------------+----------------+
159 |     TEST CASE      |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
160 +--------------------+------------------+----------------------+------------------+----------------+
161 |     rally_full     |     functest     |     benchmarking     |      91:42       |      PASS      |
162 |     rally_jobs     |     functest     |     benchmarking     |      25:24       |      PASS      |
163 |        vmtp        |     functest     |     benchmarking     |      16:47       |      PASS      |
164 |       shaker       |     functest     |     benchmarking     |      23:43       |      PASS      |
165 +--------------------+------------------+----------------------+------------------+----------------+
166 ```
167
168 ## Run benchmarking CNTT suite
169
170 ```bash
171 sudo docker run --env-file env \
172     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
173     -v $(pwd)/images:/home/opnfv/functest/images \
174     opnfv/functest-benchmarking-cntt:jerma
175 ```
176
177 ```
178 +-------------------------+------------------+----------------------+------------------+----------------+
179 |        TEST CASE        |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
180 +-------------------------+------------------+----------------------+------------------+----------------+
181 |     rally_full_cntt     |     functest     |     benchmarking     |      86:41       |      PASS      |
182 |     rally_jobs_cntt     |     functest     |     benchmarking     |      19:16       |      PASS      |
183 |           vmtp          |     functest     |     benchmarking     |      17:05       |      PASS      |
184 |          shaker         |     functest     |     benchmarking     |      23:33       |      PASS      |
185 +-------------------------+------------------+----------------------+------------------+----------------+
186 ```
187
188 ## Run vnf suite
189
190 ```bash
191 sudo docker run --env-file env \
192     -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
193     -v $(pwd)/images:/home/opnfv/functest/images \
194     opnfv/functest-vnf:jerma
195 ```
196
197 ```
198 +----------------------+------------------+--------------+------------------+----------------+
199 |      TEST CASE       |     PROJECT      |     TIER     |     DURATION     |     RESULT     |
200 +----------------------+------------------+--------------+------------------+----------------+
201 |       cloudify       |     functest     |     vnf      |      04:41       |      PASS      |
202 |     cloudify_ims     |     functest     |     vnf      |      21:03       |      PASS      |
203 |       heat_ims       |     functest     |     vnf      |      30:04       |      PASS      |
204 |     vyos_vrouter     |     functest     |     vnf      |      16:21       |      PASS      |
205 |       juju_epc       |     functest     |     vnf      |      28:33       |      PASS      |
206 +----------------------+------------------+--------------+------------------+----------------+
207 ```