README.md

   1 # Xtesting in a nutshell
   2
   3 Xtesting is a simple framework to assemble sparse test cases and to accelerate
   4 the adoption of continuous integration best practices. By managing all
   5 the interactions with the components (test scheduler, test results database,
   6 artifact repository), it allows the developer to work only on the test suites
   7 without diving into CI/CD.
   8
   9 It asks for a few low constraints
  10 [quickly achievable](https://www.sdxcentral.com/articles/news/opnfvs-6th-release-brings-testing-capabilities-that-orange-is-already-using/2018/05/)
  11 to verify multiple components in the same CI/CD toolchain. Even more, it brings
  12 the capability to run third-party test cases in our CI toolchains and then to
  13 also rate network functions by the coverage.
  14
  15 Please see
  16 [the Katacoda scenarios](https://www.katacoda.com/ollivier/courses/xtestingci)
  17 to try Xtesting. You will love them!
  18
  19 ## [Write your own Xtesting driver](https://www.katacoda.com/ollivier/courses/xtestingci/firstdriver)
  20
  21 Note that [running MongoDB 5.0+ requires _avx_ CPU instruction set](https://www.mongodb.com/docs/manual/administration/production-notes/#x86_64)
  22 that is usually shipped in all recent _x86_ hardware processors.
  23 Though, it may not be available in your virtualized environments.
  24 For example, Qemu _avx_ support is only available [since version 7.2](https://github.com/nodkz/mongodb-memory-server/issues/710#issuecomment-1297462935)
  25 and must be explicitly enabled (e.g. with the argument _-cpu max_).
  26
  27 You can check the presence of the _avx_ CPU instruction set on your processor
  28 with the following command.
  29 ```bash
  30 grep '^processor\|^flags.* avx' /proc/cpuinfo
  31 ```
  32
  33 ### dump all the following files in an empty dir
  34
  35 weather.py
  36
  37 ```python
  38 #!/usr/bin/env python
  39
  40 # pylint: disable=missing-docstring
  41
  42 import json
  43 import os
  44 import sys
  45 import time
  46
  47 import requests
  48
  49 from xtesting.core import testcase
  50
  51
  52 class Weather(testcase.TestCase):
  53
  54     url = "https://samples.openweathermap.org/data/2.5/weather"
  55     city_name = "London,uk"
  56     app_key = "439d4b804bc8187953eb36d2a8c26a02"
  57
  58     def run(self, **kwargs):
  59         try:
  60             self.start_time = time.time()
  61             req = requests.get("{}?q={}&&appid={}".format(
  62                 self.url, self.city_name, self.app_key))
  63             req.raise_for_status()
  64             data = req.json()
  65             os.makedirs(self.res_dir, exist_ok=True)
  66             with open('{}/dump.txt'.format(self.res_dir), 'w+') as report:
  67                 json.dump(data, report, indent=4, sort_keys=True)
  68             for key in kwargs:
  69                 if data["main"][key] > kwargs[key]:
  70                     self.result = self.result + 100/len(kwargs)
  71             self.stop_time = time.time()
  72         except Exception:  # pylint: disable=broad-except
  73             print("Unexpected error:", sys.exc_info()[0])
  74             self.result = 0
  75             self.stop_time = time.time()
  76 ```
  77
  78 setup.py
  79
  80 ```python
  81 #!/usr/bin/env python
  82
  83 # pylint: disable=missing-docstring
  84
  85 import setuptools
  86
  87 setuptools.setup(
  88     setup_requires=['pbr>=2.0.0'],
  89     pbr=True)
  90 ```
  91
  92 setup.cfg
  93
  94 ```
  95 [metadata]
  96 name = weather
  97 version = 1
  98
  99 [files]
 100 packages = .
 101
 102 [entry_points]
 103 xtesting.testcase =
 104     weather = weather:Weather
 105 ```
 106
 107 requirements.txt
 108
 109 ```
 110 xtesting
 111 requests!=2.20.0,!=2.24.0 # Apache-2.0
 112 ```
 113
 114 testcases.yaml
 115
 116 ```yaml
 117 ---
 118 tiers:
 119     -
 120         name: simple
 121         order: 0
 122         description: ''
 123         testcases:
 124             -
 125                 case_name: humidity
 126                 project_name: weather
 127                 criteria: 100
 128                 blocking: true
 129                 clean_flag: false
 130                 description: ''
 131                 run:
 132                     name: weather
 133                     args:
 134                         humidity: 80
 135             -
 136                 case_name: pressure
 137                 project_name: weather
 138                 criteria: 100
 139                 blocking: true
 140                 clean_flag: false
 141                 description: ''
 142                 run:
 143                     name: weather
 144                     args:
 145                         pressure: 1000
 146             -
 147                 case_name: temp
 148                 project_name: weather
 149                 criteria: 100
 150                 blocking: true
 151                 clean_flag: false
 152                 description: ''
 153                 run:
 154                     name: weather
 155                     args:
 156                         temp: 280
 157     -
 158         name: combined
 159         order: 1
 160         description: ''
 161         testcases:
 162             -
 163                 case_name: half
 164                 project_name: weather
 165                 criteria: 50
 166                 blocking: true
 167                 clean_flag: false
 168                 description: ''
 169                 run:
 170                     name: weather
 171                     args:
 172                         humidity: 90
 173                         pressure: 1000
 174                         temp: 280
 175 ```
 176
 177 Dockerfile
 178
 179 ```
 180 FROM alpine:3.18
 181
 182 ADD . /src/
 183 RUN apk --no-cache add --update python3 py3-pip py3-wheel git py3-lxml && \
 184     git init /src && pip3 install /src
 185 COPY testcases.yaml /etc/xtesting/testcases.yaml
 186 CMD ["run_tests", "-t", "all"]
 187 ```
 188
 189 site.yml
 190
 191 ```yaml
 192 ---
 193 - hosts:
 194     - 127.0.0.1
 195   roles:
 196     - role: collivier.xtesting
 197       project: weather
 198       registry_deploy: true
 199       repo: 127.0.0.1
 200       dport: 5000
 201       suites:
 202         - container: weather
 203           tests:
 204             - humidity
 205             - pressure
 206             - temp
 207             - half
 208 ```
 209
 210 ### make world
 211
 212 Deploy your own Xtesting toolchain
 213
 214 ```bash
 215 virtualenv xtesting -p python3 --system-site-packages
 216 . xtesting/bin/activate
 217 pip install ansible
 218 ansible-galaxy install collivier.xtesting
 219 ansible-galaxy collection install ansible.posix community.general community.grafana \
 220     community.kubernetes community.docker community.postgresql
 221 ansible-playbook site.yml
 222 deactivate
 223 rm -r xtesting
 224 ```
 225
 226 Build your container
 227
 228 ```bash
 229 sudo docker build -t 127.0.0.1:5000/weather .
 230 ```
 231
 232 Publish your container on your local registry
 233
 234 ```bash
 235 sudo docker push 127.0.0.1:5000/weather
 236 ```
 237
 238 ### play
 239
 240 Jenkins is accessible via http://127.0.0.1:8080 and you can identify yourself
 241 as admin to be allowed to trigger a build:
 242 - login: admin
 243 - password: admin
 244
 245 The default Jenkins view lists all the Jenkins jobs. You can easily find your
 246 main job, weather-latest-daily, via the Jenkins view named weather.
 247
 248 You're ready to start a new build of weather-latest-daily without changing
 249 the default parameters.
 250
 251 The test case is executed after a few seconds and all the test outputs are
 252 accessible via the console icons. If you open the
 253 weather-127_0_0_1-weather-latest-humidity-run, you will first read:
 254 - the test output highlighting its status
 255 - a link to the test database where its results are stored
 256 - a couple of links to its artifacts automatically published
 257
 258 A zip file dumping all test campaign data is printed in the
 259 weather-latest-zip console.
 260
 261 ### That's all folks!