README.md

   1 # Functest
   2
   3 Network virtualization has dramatically modified our architectures which asks
   4 for more automation and powerful testing tools like Functest, a collection of
   5 state-of-the-art virtual infrastructure test suites, including automatic VNF
   6 testing (cf.
   7 [[1]](https://www.linuxfoundation.org/press-release/2019/05/opnfv-hunter-delivers-test-tools-ci-cd-framework-to-enable-common-nfvi-for-verifying-vnfs/)).
   8
   9 In context of OPNFV, Functest verifies any kind of OpenStack and Kubernetes
  10 deployments including production environments. It conforms to upstream rules
  11 and integrates smoothly lots of the test cases available in the opensource
  12 market. It includes about 3000+ functional tests and 3 hours upstream API and
  13 dataplane benchmarks. It’s completed by Virtual Network Function deployments
  14 and testing (vIMS, vRouter and vEPC) to ensure that the platforms meet Network
  15 Functions Virtualization requirements. Raspberry PI is also supported to verify
  16 datacenters as the lowest cost (50 euros hardware and software included).
  17
  18 | Functest releases | Kubernetes releases           |
  19 |-------------------|-------------------------------|
  20 | v1.22             | v1.22                         |
  21 | v1.23             | v1.23                         |
  22 | v1.24             | v1.24                         |
  23 | v1.25             | v1.25                         |
  24 | **Master**        | **v1.26.0-alpha.2 (rolling)** |
  25
  26 ## Prepare your environment
  27
  28 cat env
  29 ```
  30 DEPLOY_SCENARIO=k8s-XXX
  31 ```
  32
  33 ## Run healthcheck suite
  34
  35 ```bash
  36 sudo docker run --env-file env \
  37     -v $(pwd)/config:/root/.kube/config \
  38     opnfv/functest-kubernetes-healthcheck
  39 ```
  40
  41 ```
  42 +-------------------+------------------+---------------------+------------------+----------------+
  43 |     TEST CASE     |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
  44 +-------------------+------------------+---------------------+------------------+----------------+
  45 |     k8s_quick     |     functest     |     healthcheck     |      00:18       |      PASS      |
  46 |     k8s_smoke     |     functest     |     healthcheck     |      01:14       |      PASS      |
  47 +-------------------+------------------+---------------------+------------------+----------------+
  48 ```
  49
  50 ## Run smoke suite
  51
  52 ```bash
  53 sudo docker run --env-file env \
  54     -v $(pwd)/config:/root/.kube/config \
  55     opnfv/functest-kubernetes-smoke
  56 ```
  57
  58 ```
  59 +----------------------------------+------------------+---------------+------------------+----------------+
  60 |            TEST CASE             |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
  61 +----------------------------------+------------------+---------------+------------------+----------------+
  62 |        xrally_kubernetes         |     functest     |     smoke     |      13:00       |      PASS      |
  63 |         k8s_conformance          |     functest     |     smoke     |      16:31       |      PASS      |
  64 |      k8s_conformance_serial      |     functest     |     smoke     |      15:34       |      PASS      |
  65 |        sig_api_machinery         |     functest     |     smoke     |      09:01       |      PASS      |
  66 |     sig_api_machinery_serial     |     functest     |     smoke     |      01:24       |      PASS      |
  67 |             sig_apps             |     functest     |     smoke     |      03:45       |      PASS      |
  68 |         sig_apps_serial          |     functest     |     smoke     |      00:31       |      PASS      |
  69 |             sig_auth             |     functest     |     smoke     |      09:04       |      PASS      |
  70 |      sig_cluster_lifecycle       |     functest     |     smoke     |      00:26       |      PASS      |
  71 |       sig_instrumentation        |     functest     |     smoke     |      00:03       |      PASS      |
  72 |           sig_network            |     functest     |     smoke     |      05:43       |      PASS      |
  73 |             sig_node             |     functest     |     smoke     |      28:03       |      PASS      |
  74 |      sig_scheduling_serial       |     functest     |     smoke     |      08:03       |      PASS      |
  75 |           sig_storage            |     functest     |     smoke     |      09:17       |      PASS      |
  76 |        sig_storage_serial        |     functest     |     smoke     |      02:40       |      PASS      |
  77 +----------------------------------+------------------+---------------+------------------+----------------+
  78 ```
  79
  80 ## Run security suite
  81
  82 ```bash
  83 sudo docker run --env-file env \
  84     -v $(pwd)/config:/root/.kube/config \
  85     opnfv/functest-kubernetes-security
  86 ```
  87
  88 ```
  89 +---------------------------+------------------+------------------+------------------+----------------+
  90 |         TEST CASE         |     PROJECT      |       TIER       |     DURATION     |     RESULT     |
  91 +---------------------------+------------------+------------------+------------------+----------------+
  92 |        kube_hunter        |     functest     |     security     |      00:19       |      PASS      |
  93 |     kube_bench_master     |     functest     |     security     |      00:02       |      PASS      |
  94 |      kube_bench_node      |     functest     |     security     |      00:01       |      PASS      |
  95 +---------------------------+------------------+------------------+------------------+----------------+
  96 ```
  97
  98 ## Run benchmarking suite
  99
 100 ```bash
 101 sudo docker run --env-file env \
 102     -v $(pwd)/config:/root/.kube/config \
 103     opnfv/functest-kubernetes-benchmarking
 104 ```
 105
 106 ```
 107 +--------------------------------+------------------+----------------------+------------------+----------------+
 108 |           TEST CASE            |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
 109 +--------------------------------+------------------+----------------------+------------------+----------------+
 110 |     xrally_kubernetes_full     |     functest     |     benchmarking     |      33:07       |      PASS      |
 111 |            netperf             |     functest     |     benchmarking     |      40:16       |      PASS      |
 112 +--------------------------------+------------------+----------------------+------------------+----------------+
 113 ```
 114
 115 ## Run cnf suite
 116
 117 ```bash
 118 sudo docker run --env-file env \
 119     -v $(pwd)/config:/root/.kube/config \
 120     opnfv/functest-kubernetes-cnf
 121 ```
 122
 123 ```
 124 +-----------------------+------------------+--------------+------------------+----------------+
 125 |       TEST CASE       |     PROJECT      |     TIER     |     DURATION     |     RESULT     |
 126 +-----------------------+------------------+--------------+------------------+----------------+
 127 |        k8s_vims       |     functest     |     cnf      |      09:06       |      PASS      |
 128 |       helm_vims       |     functest     |     cnf      |      08:54       |      PASS      |
 129 |     cnf_testsuite     |     functest     |     cnf      |      16:47       |      PASS      |
 130 +-----------------------+------------------+--------------+------------------+----------------+
 131 ```
 132
 133
 134 ## Use on air gap environments (no access to Internet)
 135
 136 To test a Kubernetes without access to Internet, repository mirrors needs to be
 137 provided.
 138
 139 Currently, all tests supports this feature except cnf conformance.
 140
 141 There's two ways for providing the repository mirrors:
 142
 143 - Give an environment variable (`MIRROR_REPO`) which gives a repository with
 144   all needed images.
 145 - Gives an environment variable per needed repo:
 146   - `DOCKERHUB_REPO` for DockerHub repository (`docker.io`)
 147   - `GCR_REPO` for Google Cloud repository (`gcr.io`)
 148   - `K8S_GCR_REPO` for Kubernetes repository (`k8s.gcr.io`)
 149   - `QUAY_REPO` for Quay repository (`quay.io`)
 150
 151 All needed images are given in
 152 [functest-kubernetes/ci/images.txt](functest-kubernetes/ci/images.txt)
 153
 154 For e2e tests, `docker.io` is hardcoded. it does mean that you'll have to set up
 155 a mirror on docker. An example on how to set it up on docker daemon is provided
 156 here:
 157 [daemon-configuration-file](
 158 https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file)