README.md

   1 # Functest
   2
   3 Network virtualization has dramatically modified our architectures which asks
   4 for more automation and powerful testing tools like Functest, a collection of
   5 state-of-the-art virtual infrastructure test suites, including automatic VNF
   6 testing (cf.
   7 [[1]](https://www.linuxfoundation.org/press-release/2019/05/opnfv-hunter-delivers-test-tools-ci-cd-framework-to-enable-common-nfvi-for-verifying-vnfs/)).
   8
   9 In context of OPNFV, Functest verifies any kind of OpenStack and Kubernetes
  10 deployments including production environments. It conforms to upstream rules
  11 and integrates smoothly lots of the test cases available in the opensource
  12 market. It includes about 3000+ functional tests and 3 hours upstream API and
  13 dataplane benchmarks. It’s completed by Virtual Network Function deployments
  14 and testing (vIMS, vRouter and vEPC) to ensure that the platforms meet Network
  15 Functions Virtualization requirements. Raspberry PI is also supported to verify
  16 datacenters as the lowest cost (50 euros hardware and software included).
  17
  18 | Functest releases | Kubernetes releases           |
  19 |-------------------|-------------------------------|
  20 | v1.22             | v1.22                         |
  21 | v1.23             | v1.23                         |
  22 | v1.24             | v1.24                         |
  23 | v1.25             | v1.25                         |
  24 | v1.26             | v1.26                         |
  25 | v1.27             | v1.27                         |
  26 | v1.28             | v1.28                         |
  27 | **v1.29**         | **v1.29**                     |
  28 | Master            | v1.30.0-alpha.0 (rolling)     |
  29
  30 ## Prepare your environment
  31
  32 cat env
  33 ```
  34 DEPLOY_SCENARIO=k8s-XXX
  35 ```
  36
  37 ## Run healthcheck suite
  38
  39 ```bash
  40 sudo docker run --env-file env \
  41     -v $(pwd)/config:/home/xtesting/.kube/config \
  42     opnfv/functest-kubernetes-healthcheck:v1.29
  43 ```
  44
  45 ```
  46 +-------------------+------------------+---------------------+------------------+----------------+
  47 |     TEST CASE     |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
  48 +-------------------+------------------+---------------------+------------------+----------------+
  49 |     k8s_quick     |     functest     |     healthcheck     |      00:18       |      PASS      |
  50 |     k8s_smoke     |     functest     |     healthcheck     |      01:14       |      PASS      |
  51 +-------------------+------------------+---------------------+------------------+----------------+
  52 ```
  53
  54 ## Run smoke suite
  55
  56 ```bash
  57 sudo docker run --env-file env \
  58     -v $(pwd)/config:/home/xtesting/.kube/config \
  59     opnfv/functest-kubernetes-smoke:v1.29
  60 ```
  61
  62 ```
  63 +----------------------------------+------------------+---------------+------------------+----------------+
  64 |            TEST CASE             |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
  65 +----------------------------------+------------------+---------------+------------------+----------------+
  66 |        xrally_kubernetes         |     functest     |     smoke     |      13:00       |      PASS      |
  67 |         k8s_conformance          |     functest     |     smoke     |      16:31       |      PASS      |
  68 |      k8s_conformance_serial      |     functest     |     smoke     |      15:34       |      PASS      |
  69 |        sig_api_machinery         |     functest     |     smoke     |      09:01       |      PASS      |
  70 |     sig_api_machinery_serial     |     functest     |     smoke     |      01:24       |      PASS      |
  71 |             sig_apps             |     functest     |     smoke     |      03:45       |      PASS      |
  72 |         sig_apps_serial          |     functest     |     smoke     |      00:31       |      PASS      |
  73 |             sig_auth             |     functest     |     smoke     |      09:04       |      PASS      |
  74 |      sig_cluster_lifecycle       |     functest     |     smoke     |      00:26       |      PASS      |
  75 |       sig_instrumentation        |     functest     |     smoke     |      00:03       |      PASS      |
  76 |           sig_network            |     functest     |     smoke     |      05:43       |      PASS      |
  77 |             sig_node             |     functest     |     smoke     |      28:03       |      PASS      |
  78 |      sig_scheduling_serial       |     functest     |     smoke     |      08:03       |      PASS      |
  79 |           sig_storage            |     functest     |     smoke     |      09:17       |      PASS      |
  80 |        sig_storage_serial        |     functest     |     smoke     |      02:40       |      PASS      |
  81 +----------------------------------+------------------+---------------+------------------+----------------+
  82 ```
  83
  84 ## Run security suite
  85
  86 ```bash
  87 sudo docker run --env-file env \
  88     -v $(pwd)/config:/home/xtesting/.kube/config \
  89     opnfv/functest-kubernetes-security:v1.29
  90 ```
  91
  92 ```
  93 +---------------------------+------------------+------------------+------------------+----------------+
  94 |         TEST CASE         |     PROJECT      |       TIER       |     DURATION     |     RESULT     |
  95 +---------------------------+------------------+------------------+------------------+----------------+
  96 |        kube_hunter        |     functest     |     security     |      00:19       |      PASS      |
  97 |     kube_bench_master     |     functest     |     security     |      00:02       |      PASS      |
  98 |      kube_bench_node      |     functest     |     security     |      00:01       |      PASS      |
  99 +---------------------------+------------------+------------------+------------------+----------------+
 100 ```
 101
 102 ## Run benchmarking suite
 103
 104 ```bash
 105 sudo docker run --env-file env \
 106     -v $(pwd)/config:/home/xtesting/.kube/config \
 107     opnfv/functest-kubernetes-benchmarking:v1.29
 108 ```
 109
 110 ```
 111 +--------------------------------+------------------+----------------------+------------------+----------------+
 112 |           TEST CASE            |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
 113 +--------------------------------+------------------+----------------------+------------------+----------------+
 114 |     xrally_kubernetes_full     |     functest     |     benchmarking     |      33:07       |      PASS      |
 115 |            netperf             |     functest     |     benchmarking     |      40:16       |      PASS      |
 116 +--------------------------------+------------------+----------------------+------------------+----------------+
 117 ```
 118
 119 ## Run cnf suite
 120
 121 ```bash
 122 sudo docker run --env-file env \
 123     -v $(pwd)/config:/home/xtesting/.kube/config \
 124     opnfv/functest-kubernetes-cnf:v1.29
 125 ```
 126
 127 ```
 128 +-----------------------+------------------+--------------+------------------+----------------+
 129 |       TEST CASE       |     PROJECT      |     TIER     |     DURATION     |     RESULT     |
 130 +-----------------------+------------------+--------------+------------------+----------------+
 131 |        k8s_vims       |     functest     |     cnf      |      09:06       |      PASS      |
 132 |       helm_vims       |     functest     |     cnf      |      08:54       |      PASS      |
 133 |     cnf_testsuite     |     functest     |     cnf      |      16:47       |      PASS      |
 134 +-----------------------+------------------+--------------+------------------+----------------+
 135 ```
 136
 137
 138 ## Use on air gap environments (no access to Internet)
 139
 140 To test a Kubernetes without access to Internet, repository mirrors needs to be
 141 provided.
 142
 143 Currently, all tests supports this feature except cnf conformance.
 144
 145 There's two ways for providing the repository mirrors:
 146
 147 - Give an environment variable (`MIRROR_REPO`) which gives a repository with
 148   all needed images.
 149 - Gives an environment variable per needed repo:
 150   - `DOCKERHUB_REPO` for DockerHub repository (`docker.io`)
 151   - `GCR_REPO` for Google Cloud repository (`gcr.io`)
 152   - `K8S_GCR_REPO` for Kubernetes repository (`k8s.gcr.io`)
 153   - `QUAY_REPO` for Quay repository (`quay.io`)
 154
 155 All needed images are given in
 156 [functest_kubernetes/ci/images.txt](functest_kubernetes/ci/images.txt)
 157
 158 For e2e tests, `docker.io` is hardcoded. it does mean that you'll have to set up
 159 a mirror on docker. An example on how to set it up on docker daemon is provided
 160 here:
 161 [daemon-configuration-file](
 162 https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file)