README.md

   1 # Functest
   2
   3 Network virtualization has dramatically modified our architectures which asks
   4 for more automation and powerful testing tools like Functest, a collection of
   5 state-of-the-art virtual infrastructure test suites, including automatic VNF
   6 testing (cf.
   7 [[1]](https://www.linuxfoundation.org/press-release/2019/05/opnfv-hunter-delivers-test-tools-ci-cd-framework-to-enable-common-nfvi-for-verifying-vnfs/)).
   8
   9 In context of OPNFV, Functest verifies any kind of OpenStack and Kubernetes
  10 deployments including production environments. It conforms to upstream rules
  11 and integrates smoothly lots of the test cases available in the opensource
  12 market. It includes about 3000+ functional tests and 3 hours upstream API and
  13 dataplane benchmarks. It’s completed by Virtual Network Function deployments
  14 and testing (vIMS, vRouter and vEPC) to ensure that the platforms meet Network
  15 Functions Virtualization requirements. Raspberry PI is also supported to verify
  16 datacenters as the lowest cost (50 euros hardware and software included).
  17
  18 | Functest releases | Kubernetes releases           |
  19 |-------------------|-------------------------------|
  20 | v1.22             | v1.22                         |
  21 | v1.23             | v1.23                         |
  22 | v1.24             | v1.24                         |
  23 | v1.25             | v1.25                         |
  24 | v1.26             | v1.26                         |
  25 | **Master**        | **v1.27.0-alpha.0 (rolling)** |
  26
  27 ## Prepare your environment
  28
  29 cat env
  30 ```
  31 DEPLOY_SCENARIO=k8s-XXX
  32 ```
  33
  34 ## Run healthcheck suite
  35
  36 ```bash
  37 sudo docker run --env-file env \
  38     -v $(pwd)/config:/home/xtesting/.kube/config \
  39     opnfv/functest-kubernetes-healthcheck
  40 ```
  41
  42 ```
  43 +-------------------+------------------+---------------------+------------------+----------------+
  44 |     TEST CASE     |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
  45 +-------------------+------------------+---------------------+------------------+----------------+
  46 |     k8s_quick     |     functest     |     healthcheck     |      00:18       |      PASS      |
  47 |     k8s_smoke     |     functest     |     healthcheck     |      01:14       |      PASS      |
  48 +-------------------+------------------+---------------------+------------------+----------------+
  49 ```
  50
  51 ## Run smoke suite
  52
  53 ```bash
  54 sudo docker run --env-file env \
  55     -v $(pwd)/config:/home/xtesting/.kube/config \
  56     opnfv/functest-kubernetes-smoke
  57 ```
  58
  59 ```
  60 +----------------------------------+------------------+---------------+------------------+----------------+
  61 |            TEST CASE             |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
  62 +----------------------------------+------------------+---------------+------------------+----------------+
  63 |        xrally_kubernetes         |     functest     |     smoke     |      13:00       |      PASS      |
  64 |         k8s_conformance          |     functest     |     smoke     |      16:31       |      PASS      |
  65 |      k8s_conformance_serial      |     functest     |     smoke     |      15:34       |      PASS      |
  66 |        sig_api_machinery         |     functest     |     smoke     |      09:01       |      PASS      |
  67 |     sig_api_machinery_serial     |     functest     |     smoke     |      01:24       |      PASS      |
  68 |             sig_apps             |     functest     |     smoke     |      03:45       |      PASS      |
  69 |         sig_apps_serial          |     functest     |     smoke     |      00:31       |      PASS      |
  70 |             sig_auth             |     functest     |     smoke     |      09:04       |      PASS      |
  71 |      sig_cluster_lifecycle       |     functest     |     smoke     |      00:26       |      PASS      |
  72 |       sig_instrumentation        |     functest     |     smoke     |      00:03       |      PASS      |
  73 |           sig_network            |     functest     |     smoke     |      05:43       |      PASS      |
  74 |             sig_node             |     functest     |     smoke     |      28:03       |      PASS      |
  75 |      sig_scheduling_serial       |     functest     |     smoke     |      08:03       |      PASS      |
  76 |           sig_storage            |     functest     |     smoke     |      09:17       |      PASS      |
  77 |        sig_storage_serial        |     functest     |     smoke     |      02:40       |      PASS      |
  78 +----------------------------------+------------------+---------------+------------------+----------------+
  79 ```
  80
  81 ## Run security suite
  82
  83 ```bash
  84 sudo docker run --env-file env \
  85     -v $(pwd)/config:/home/xtesting/.kube/config \
  86     opnfv/functest-kubernetes-security
  87 ```
  88
  89 ```
  90 +---------------------------+------------------+------------------+------------------+----------------+
  91 |         TEST CASE         |     PROJECT      |       TIER       |     DURATION     |     RESULT     |
  92 +---------------------------+------------------+------------------+------------------+----------------+
  93 |        kube_hunter        |     functest     |     security     |      00:19       |      PASS      |
  94 |     kube_bench_master     |     functest     |     security     |      00:02       |      PASS      |
  95 |      kube_bench_node      |     functest     |     security     |      00:01       |      PASS      |
  96 +---------------------------+------------------+------------------+------------------+----------------+
  97 ```
  98
  99 ## Run benchmarking suite
 100
 101 ```bash
 102 sudo docker run --env-file env \
 103     -v $(pwd)/config:/home/xtesting/.kube/config \
 104     opnfv/functest-kubernetes-benchmarking
 105 ```
 106
 107 ```
 108 +--------------------------------+------------------+----------------------+------------------+----------------+
 109 |           TEST CASE            |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
 110 +--------------------------------+------------------+----------------------+------------------+----------------+
 111 |     xrally_kubernetes_full     |     functest     |     benchmarking     |      33:07       |      PASS      |
 112 |            netperf             |     functest     |     benchmarking     |      40:16       |      PASS      |
 113 +--------------------------------+------------------+----------------------+------------------+----------------+
 114 ```
 115
 116 ## Run cnf suite
 117
 118 ```bash
 119 sudo docker run --env-file env \
 120     -v $(pwd)/config:/home/xtesting/.kube/config \
 121     opnfv/functest-kubernetes-cnf
 122 ```
 123
 124 ```
 125 +-----------------------+------------------+--------------+------------------+----------------+
 126 |       TEST CASE       |     PROJECT      |     TIER     |     DURATION     |     RESULT     |
 127 +-----------------------+------------------+--------------+------------------+----------------+
 128 |        k8s_vims       |     functest     |     cnf      |      09:06       |      PASS      |
 129 |       helm_vims       |     functest     |     cnf      |      08:54       |      PASS      |
 130 |     cnf_testsuite     |     functest     |     cnf      |      16:47       |      PASS      |
 131 +-----------------------+------------------+--------------+------------------+----------------+
 132 ```
 133
 134
 135 ## Use on air gap environments (no access to Internet)
 136
 137 To test a Kubernetes without access to Internet, repository mirrors needs to be
 138 provided.
 139
 140 Currently, all tests supports this feature except cnf conformance.
 141
 142 There's two ways for providing the repository mirrors:
 143
 144 - Give an environment variable (`MIRROR_REPO`) which gives a repository with
 145   all needed images.
 146 - Gives an environment variable per needed repo:
 147   - `DOCKERHUB_REPO` for DockerHub repository (`docker.io`)
 148   - `GCR_REPO` for Google Cloud repository (`gcr.io`)
 149   - `K8S_GCR_REPO` for Kubernetes repository (`k8s.gcr.io`)
 150   - `QUAY_REPO` for Quay repository (`quay.io`)
 151
 152 All needed images are given in
 153 [functest_kubernetes/ci/images.txt](functest_kubernetes/ci/images.txt)
 154
 155 For e2e tests, `docker.io` is hardcoded. it does mean that you'll have to set up
 156 a mirror on docker. An example on how to set it up on docker daemon is provided
 157 here:
 158 [daemon-configuration-file](
 159 https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file)