Enforce self.details as a collection
[functest-xtesting.git] / README.md
1 # Xtesting in a nutshell
2
3 Xtesting is a simple framework to assemble sparse test cases and to accelerate
4 the adoption of continuous integration best practices. By managing all
5 the interactions with the components (test scheduler, test results database,
6 artifact repository), it allows the developer to work only on the test suites
7 without diving into CI/CD.
8
9 It asks for a few low constraints
10 [quickly achievable](https://www.sdxcentral.com/articles/news/opnfvs-6th-release-brings-testing-capabilities-that-orange-is-already-using/2018/05/)
11 to verify multiple components in the same CI/CD toolchain. Even more, it brings
12 the capability to run third-party test cases in our CI toolchains and then to
13 also rate network functions by the coverage.
14
15 Please see
16 [the Katacoda scenarios](https://www.katacoda.com/ollivier/courses/xtestingci)
17 to try Xtesting. You will love them!
18
19 ## [Write your own Xtesting driver](https://www.katacoda.com/ollivier/courses/xtestingci/firstdriver)
20
21 Note that [running MongoDB 5.0+ requires _avx_ CPU instruction set](https://www.mongodb.com/docs/manual/administration/production-notes/#x86_64)
22 that is usually shipped in all recent _x86_ hardware processors.
23 Though, it may not be available in your virtualized environments.
24 For example, Qemu _avx_ support is only available [since version 7.2](https://github.com/nodkz/mongodb-memory-server/issues/710#issuecomment-1297462935)
25 and must be explicitly enabled (e.g. with the argument _-cpu max_).
26
27 You can check the presence of the _avx_ CPU instruction set on your processor
28 with the following command.
29 ```bash
30 grep '^processor\|^flags.* avx' /proc/cpuinfo
31 ```
32
33 ### dump all the following files in an empty dir
34
35 weather.py
36
37 ```python
38 #!/usr/bin/env python
39
40 # pylint: disable=missing-docstring
41
42 import json
43 import os
44 import sys
45 import time
46
47 import requests
48
49 from xtesting.core import testcase
50
51
52 class Weather(testcase.TestCase):
53
54     url = "https://samples.openweathermap.org/data/2.5/weather"
55     city_name = "London,uk"
56     app_key = "439d4b804bc8187953eb36d2a8c26a02"
57
58     def run(self, **kwargs):
59         try:
60             self.start_time = time.time()
61             req = requests.get("{}?q={}&&appid={}".format(
62                 self.url, self.city_name, self.app_key))
63             req.raise_for_status()
64             data = req.json()
65             os.makedirs(self.res_dir, exist_ok=True)
66             with open('{}/dump.txt'.format(self.res_dir), 'w+') as report:
67                 json.dump(data, report, indent=4, sort_keys=True)
68             for key in kwargs:
69                 if data["main"][key] > kwargs[key]:
70                     self.result = self.result + 100/len(kwargs)
71             self.stop_time = time.time()
72         except Exception:  # pylint: disable=broad-except
73             print("Unexpected error:", sys.exc_info()[0])
74             self.result = 0
75             self.stop_time = time.time()
76 ```
77
78 setup.py
79
80 ```python
81 #!/usr/bin/env python
82
83 # pylint: disable=missing-docstring
84
85 import setuptools
86
87 setuptools.setup(
88     setup_requires=['pbr>=2.0.0'],
89     pbr=True)
90 ```
91
92 setup.cfg
93
94 ```
95 [metadata]
96 name = weather
97 version = 1
98
99 [files]
100 packages = .
101
102 [entry_points]
103 xtesting.testcase =
104     weather = weather:Weather
105 ```
106
107 requirements.txt
108
109 ```
110 xtesting
111 requests!=2.20.0,!=2.24.0 # Apache-2.0
112 ```
113
114 testcases.yaml
115
116 ```yaml
117 ---
118 tiers:
119     -
120         name: simple
121         order: 0
122         description: ''
123         testcases:
124             -
125                 case_name: humidity
126                 project_name: weather
127                 criteria: 100
128                 blocking: true
129                 clean_flag: false
130                 description: ''
131                 run:
132                     name: weather
133                     args:
134                         humidity: 80
135             -
136                 case_name: pressure
137                 project_name: weather
138                 criteria: 100
139                 blocking: true
140                 clean_flag: false
141                 description: ''
142                 run:
143                     name: weather
144                     args:
145                         pressure: 1000
146             -
147                 case_name: temp
148                 project_name: weather
149                 criteria: 100
150                 blocking: true
151                 clean_flag: false
152                 description: ''
153                 run:
154                     name: weather
155                     args:
156                         temp: 280
157     -
158         name: combined
159         order: 1
160         description: ''
161         testcases:
162             -
163                 case_name: half
164                 project_name: weather
165                 criteria: 50
166                 blocking: true
167                 clean_flag: false
168                 description: ''
169                 run:
170                     name: weather
171                     args:
172                         humidity: 90
173                         pressure: 1000
174                         temp: 280
175 ```
176
177 Dockerfile
178
179 ```
180 FROM alpine:3.18
181
182 ADD . /src/
183 RUN apk --no-cache add --update python3 py3-pip py3-wheel git py3-lxml && \
184     git init /src && pip3 install /src
185 COPY testcases.yaml /etc/xtesting/testcases.yaml
186 CMD ["run_tests", "-t", "all"]
187 ```
188
189 site.yml
190
191 ```yaml
192 ---
193 - hosts:
194     - 127.0.0.1
195   roles:
196     - role: collivier.xtesting
197       project: weather
198       registry_deploy: true
199       repo: 127.0.0.1
200       dport: 5000
201       suites:
202         - container: weather
203           tests:
204             - humidity
205             - pressure
206             - temp
207             - half
208 ```
209
210 ### make world
211
212 Deploy your own Xtesting toolchain
213
214 ```bash
215 virtualenv xtesting -p python3 --system-site-packages
216 . xtesting/bin/activate
217 pip install ansible
218 ansible-galaxy install collivier.xtesting
219 ansible-galaxy collection install ansible.posix community.general community.grafana \
220     community.kubernetes community.docker community.postgresql
221 ansible-playbook site.yml
222 deactivate
223 rm -r xtesting
224 ```
225
226 Build your container
227
228 ```bash
229 sudo docker build -t 127.0.0.1:5000/weather .
230 ```
231
232 Publish your container on your local registry
233
234 ```bash
235 sudo docker push 127.0.0.1:5000/weather
236 ```
237
238 ### play
239
240 Jenkins is accessible via http://127.0.0.1:8080 and you can identify yourself
241 as admin to be allowed to trigger a build:
242 - login: admin
243 - password: admin
244
245 The default Jenkins view lists all the Jenkins jobs. You can easily find your
246 main job, weather-latest-daily, via the Jenkins view named weather.
247
248 You're ready to start a new build of weather-latest-daily without changing
249 the default parameters.
250
251 The test case is executed after a few seconds and all the test outputs are
252 accessible via the console icons. If you open the
253 weather-127_0_0_1-weather-latest-humidity-run, you will first read:
254 - the test output highlighting its status
255 - a link to the test database where its results are stored
256 - a couple of links to its artifacts automatically published
257
258 A zip file dumping all test campaign data is printed in the
259 weather-latest-zip console.
260
261 ### That's all folks!