README.md

   1 # Functest
   2
   3 Network virtualization has dramatically modified our architectures which asks
   4 for more automation and powerful testing tools like Functest, a collection of
   5 state-of-the-art virtual infrastructure test suites, including automatic VNF
   6 testing (cf.
   7 [[1]](https://www.linuxfoundation.org/press-release/2019/05/opnfv-hunter-delivers-test-tools-ci-cd-framework-to-enable-common-nfvi-for-verifying-vnfs/)).
   8
   9 In context of OPNFV, Functest verifies any kind of OpenStack and Kubernetes
  10 deployments including production environments. It conforms to upstream rules
  11 and integrates smoothly lots of the test cases available in the opensource
  12 market. It includes about 3000+ functional tests and 3 hours upstream API and
  13 dataplane benchmarks. It’s completed by Virtual Network Function deployments
  14 and testing (vIMS, vRouter and vEPC) to ensure that the platforms meet Network
  15 Functions Virtualization requirements. Raspberry PI is also supported to verify
  16 datacenters as the lowest cost (50 euros hardware and software included).
  17
  18 | Functest releases | Kubernetes releases           |
  19 |-------------------|-------------------------------|
  20 | v1.22             | v1.22                         |
  21 | v1.23             | v1.23                         |
  22 | v1.24             | v1.24                         |
  23 | v1.25             | v1.25                         |
  24 | v1.26             | v1.26                         |
  25 | v1.27             | v1.27                         |
  26 | **v1.28**         | **v1.28**                     |
  27 | Master            | v1.30.0-alpha.0 (rolling)     |
  28
  29 ## Prepare your environment
  30
  31 cat env
  32 ```
  33 DEPLOY_SCENARIO=k8s-XXX
  34 ```
  35
  36 ## Run healthcheck suite
  37
  38 ```bash
  39 sudo docker run --env-file env \
  40     -v $(pwd)/config:/home/xtesting/.kube/config \
  41     opnfv/functest-kubernetes-healthcheck:v1.28
  42 ```
  43
  44 ```
  45 +-------------------+------------------+---------------------+------------------+----------------+
  46 |     TEST CASE     |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
  47 +-------------------+------------------+---------------------+------------------+----------------+
  48 |     k8s_quick     |     functest     |     healthcheck     |      00:18       |      PASS      |
  49 |     k8s_smoke     |     functest     |     healthcheck     |      01:14       |      PASS      |
  50 +-------------------+------------------+---------------------+------------------+----------------+
  51 ```
  52
  53 ## Run smoke suite
  54
  55 ```bash
  56 sudo docker run --env-file env \
  57     -v $(pwd)/config:/home/xtesting/.kube/config \
  58     opnfv/functest-kubernetes-smoke:v1.28
  59 ```
  60
  61 ```
  62 +----------------------------------+------------------+---------------+------------------+----------------+
  63 |            TEST CASE             |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
  64 +----------------------------------+------------------+---------------+------------------+----------------+
  65 |        xrally_kubernetes         |     functest     |     smoke     |      13:00       |      PASS      |
  66 |         k8s_conformance          |     functest     |     smoke     |      16:31       |      PASS      |
  67 |      k8s_conformance_serial      |     functest     |     smoke     |      15:34       |      PASS      |
  68 |        sig_api_machinery         |     functest     |     smoke     |      09:01       |      PASS      |
  69 |     sig_api_machinery_serial     |     functest     |     smoke     |      01:24       |      PASS      |
  70 |             sig_apps             |     functest     |     smoke     |      03:45       |      PASS      |
  71 |         sig_apps_serial          |     functest     |     smoke     |      00:31       |      PASS      |
  72 |             sig_auth             |     functest     |     smoke     |      09:04       |      PASS      |
  73 |      sig_cluster_lifecycle       |     functest     |     smoke     |      00:26       |      PASS      |
  74 |       sig_instrumentation        |     functest     |     smoke     |      00:03       |      PASS      |
  75 |           sig_network            |     functest     |     smoke     |      05:43       |      PASS      |
  76 |             sig_node             |     functest     |     smoke     |      28:03       |      PASS      |
  77 |      sig_scheduling_serial       |     functest     |     smoke     |      08:03       |      PASS      |
  78 |           sig_storage            |     functest     |     smoke     |      09:17       |      PASS      |
  79 |        sig_storage_serial        |     functest     |     smoke     |      02:40       |      PASS      |
  80 +----------------------------------+------------------+---------------+------------------+----------------+
  81 ```
  82
  83 ## Run security suite
  84
  85 ```bash
  86 sudo docker run --env-file env \
  87     -v $(pwd)/config:/home/xtesting/.kube/config \
  88     opnfv/functest-kubernetes-security:v1.28
  89 ```
  90
  91 ```
  92 +---------------------------+------------------+------------------+------------------+----------------+
  93 |         TEST CASE         |     PROJECT      |       TIER       |     DURATION     |     RESULT     |
  94 +---------------------------+------------------+------------------+------------------+----------------+
  95 |        kube_hunter        |     functest     |     security     |      00:19       |      PASS      |
  96 |     kube_bench_master     |     functest     |     security     |      00:02       |      PASS      |
  97 |      kube_bench_node      |     functest     |     security     |      00:01       |      PASS      |
  98 +---------------------------+------------------+------------------+------------------+----------------+
  99 ```
 100
 101 ## Run benchmarking suite
 102
 103 ```bash
 104 sudo docker run --env-file env \
 105     -v $(pwd)/config:/home/xtesting/.kube/config \
 106     opnfv/functest-kubernetes-benchmarking:v1.28
 107 ```
 108
 109 ```
 110 +--------------------------------+------------------+----------------------+------------------+----------------+
 111 |           TEST CASE            |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
 112 +--------------------------------+------------------+----------------------+------------------+----------------+
 113 |     xrally_kubernetes_full     |     functest     |     benchmarking     |      33:07       |      PASS      |
 114 |            netperf             |     functest     |     benchmarking     |      40:16       |      PASS      |
 115 +--------------------------------+------------------+----------------------+------------------+----------------+
 116 ```
 117
 118 ## Run cnf suite
 119
 120 ```bash
 121 sudo docker run --env-file env \
 122     -v $(pwd)/config:/home/xtesting/.kube/config \
 123     opnfv/functest-kubernetes-cnf:v1.28
 124 ```
 125
 126 ```
 127 +-----------------------+------------------+--------------+------------------+----------------+
 128 |       TEST CASE       |     PROJECT      |     TIER     |     DURATION     |     RESULT     |
 129 +-----------------------+------------------+--------------+------------------+----------------+
 130 |        k8s_vims       |     functest     |     cnf      |      09:06       |      PASS      |
 131 |       helm_vims       |     functest     |     cnf      |      08:54       |      PASS      |
 132 |     cnf_testsuite     |     functest     |     cnf      |      16:47       |      PASS      |
 133 +-----------------------+------------------+--------------+------------------+----------------+
 134 ```
 135
 136
 137 ## Use on air gap environments (no access to Internet)
 138
 139 To test a Kubernetes without access to Internet, repository mirrors needs to be
 140 provided.
 141
 142 Currently, all tests supports this feature except cnf conformance.
 143
 144 There's two ways for providing the repository mirrors:
 145
 146 - Give an environment variable (`MIRROR_REPO`) which gives a repository with
 147   all needed images.
 148 - Gives an environment variable per needed repo:
 149   - `DOCKERHUB_REPO` for DockerHub repository (`docker.io`)
 150   - `GCR_REPO` for Google Cloud repository (`gcr.io`)
 151   - `K8S_GCR_REPO` for Kubernetes repository (`k8s.gcr.io`)
 152   - `QUAY_REPO` for Quay repository (`quay.io`)
 153
 154 All needed images are given in
 155 [functest_kubernetes/ci/images.txt](functest_kubernetes/ci/images.txt)
 156
 157 For e2e tests, `docker.io` is hardcoded. it does mean that you'll have to set up
 158 a mirror on docker. An example on how to set it up on docker daemon is provided
 159 here:
 160 [daemon-configuration-file](
 161 https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file)