Most the features are stopped and Functest shouldn't host dead code.
Functest is a framework (Docker and Python modules) which can be still
reused by any third-party (OPNFV Feature or anything else) in its tree
and CI.
Change-Id: Ia8475ce295b2cdcddbfcfc5b17ea2ab9e51a8afb
Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
- amd64_dirs=""
- arm64_dirs=""
- arm_dirs="docker/benchmarking"
- - script: sudo -E bash build.sh
- env:
- - REPO="${DOCKER_USERNAME}"
- - amd64_dirs="docker/features"
- - arm64_dirs=""
- - arm_dirs=""
- - script: sudo -E bash build.sh
- env:
- - REPO="${DOCKER_USERNAME}"
- - amd64_dirs=""
- - arm64_dirs="docker/features"
- - arm_dirs=""
- - script: sudo -E bash build.sh
- env:
- - REPO="${DOCKER_USERNAME}"
- - amd64_dirs=""
- - arm64_dirs=""
- - arm_dirs="docker/features"
- script: sudo -E bash build.sh
env:
- REPO="${DOCKER_USERNAME}"
--platforms linux/amd64,linux/arm,linux/arm64 \
--template ${DOCKER_USERNAME}/functest-benchmarking:ARCH-latest \
--target ${DOCKER_USERNAME}/functest-benchmarking:latest
- - script: >
- sudo manifest-tool push from-args \
- --platforms linux/amd64,linux/arm,linux/arm64 \
- --template ${DOCKER_USERNAME}/functest-features:ARCH-latest \
- --target ${DOCKER_USERNAME}/functest-features:latest
- script: >
sudo manifest-tool push from-args \
--platforms linux/amd64,linux/arm,linux/arm64 \
- name: functest-vnf
ref_arg:
path: docker/vnf
- - name: functest-features
- ref_arg: BRANCH
- path: docker/features
suites:
- container: functest-healthcheck
tests:
- juju_epc
properties:
execution-type: SEQUENTIALLY
- - container: functest-features
- tests:
- - doctor-notification
- - functest-odl-sfc
- - barometercollectd
- - vgpu
- - stor4nfv_os
docker/healthcheck \
docker/smoke \
docker/benchmarking \
-docker/features \
docker/vnf"}
arm_dirs=${arm_dirs-"\
docker/core \
docker/tempest \
docker/healthcheck \
docker/smoke \
-docker/benchmarking \
-docker/features"}
+docker/benchmarking"}
arm64_dirs=${arm64_dirs-${amd64_dirs}}
build_opts=("--pull=true" --no-cache "--force-rm=true")
+++ /dev/null
-FROM opnfv/functest-core
-
-ARG BRANCH=master
-ARG OPENSTACK_TAG=master
-
-COPY thirdparty-requirements.txt thirdparty-requirements.txt
-RUN apk --no-cache add --update sshpass && \
- apk --no-cache add --virtual .build-deps --update \
- python3-dev build-base linux-headers libffi-dev \
- openssl-dev libjpeg-turbo-dev file && \
- wget -q -O- https://opendev.org/openstack/requirements/raw/branch/$OPENSTACK_TAG/upper-constraints.txt > upper-constraints.txt && \
- sed -i -E /^tempest==+.*$/d upper-constraints.txt && \
- case $(uname -m) in aarch*|arm*) sed -i -E /^PyNaCl=/d upper-constraints.txt ;; esac && \
- wget -q -O- https://git.opnfv.org/functest/plain/upper-constraints.txt?h=$BRANCH > upper-constraints.opnfv.txt && \
- sed -i -E /#egg=functest/d upper-constraints.opnfv.txt && \
- pip3 install --no-cache-dir --src /src -cupper-constraints.txt \
- -cupper-constraints.opnfv.txt \
- -rthirdparty-requirements.txt && \
- rm -r upper-constraints.txt upper-constraints.opnfv.txt thirdparty-requirements.txt && \
- apk del .build-deps
-COPY testcases.yaml /usr/lib/python3.7/site-packages/xtesting/ci/testcases.yaml
-CMD ["run_tests", "-t", "all"]
+++ /dev/null
-#!/bin/bash
-
-from="${DOCKER_REPO%/*}/functest-tempest:${DOCKER_TAG}"
-sed -i "s|^FROM.*$|FROM ${from}|" Dockerfile
-
-exit $?
+++ /dev/null
----
-tiers:
- -
- name: features
- order: 3
- description: >-
- Test suites from feature projects
- integrated in functest
- testcases:
- -
- case_name: doctor-notification
- project_name: doctor
- enabled: false
- criteria: 100
- blocking: false
- description: >-
- Test suite from Doctor project.
- dependencies:
- - INSTALLER_TYPE: '(apex)|(fuel)|(daisy)'
- - DEPLOY_SCENARIO: '^((?!fdio|nofeature).)*$'
- run:
- name: bashfeature
- args:
- cmd: 'doctor-test'
-
- -
- case_name: functest-odl-sfc
- project_name: sfc
- criteria: 100
- blocking: false
- description: >-
- Test suite for odl-sfc to test two chains with one SF and
- one chain with two SFs
- dependencies:
- - DEPLOY_SCENARIO: 'odl.*sfc'
- run:
- name: functest-odl-sfc
-
- -
- case_name: barometercollectd
- project_name: barometer
- enabled: false
- criteria: 100
- blocking: false
- description: >-
- Test suite for the Barometer project. Separate tests verify
- the proper configuration and basic functionality of all the
- collectd plugins as described in the Project Release Plan
- dependencies:
- - DEPLOY_SCENARIO: 'bar'
- run:
- name: barometercollectd
-
- -
- case_name: vgpu
- project_name: functest
- criteria: 100
- blocking: false
- description: >-
- Test suite for the OpenStack vGPU feature
- dependencies:
- - DEPLOY_SCENARIO: 'vgpu'
- run:
- name: vgpu
-
- -
- case_name: stor4nfv_os
- project_name: stor4nfv
- enabled: false
- criteria: 100
- blocking: false
- description: >-
- This tests are necessary demonstrate conformance
- of the OpenStack+Stor4NFV deployment.
- dependencies:
- - DEPLOY_SCENARIO: 'stor4nfv'
- run:
- name: stor4nfv_os
+++ /dev/null
-robotframework-httplibrary
-robotframework-requests
-robotframework-sshlibrary
-# baro_tests
-# sfc
-# stor4nfv_tests
-# doctor-tests
* heat_ims
* vyos_vrouter
* juju_epc
- * vgpu
-
-The OPNFV projects integrated into Functest framework for automation are:
-
- * doctor
- * odl-sfc
- * barometer
- * stor4nfv_os
Kubernetes
----------
* k8s_smoke
* k8s_conformance
-The OPNFV projects integrated into Functest framework for automation are:
-
- * stor4nfv
- * clover
-
Release Data
============
* https://hub.docker.com/r/opnfv/functest-healthcheck
* https://hub.docker.com/r/opnfv/functest-smoke
* https://hub.docker.com/r/opnfv/functest-benchmarking
- * https://hub.docker.com/r/opnfv/functest-features
* https://hub.docker.com/r/opnfv/functest-vnf
Functest Docker images (Kubernetes):
* https://hub.docker.com/r/opnfv/functest-kubernetes-healthcheck
* https://hub.docker.com/r/opnfv/functest-kubernetes-smoke
- * https://hub.docker.com/r/opnfv/functest-kubernetes-features
Docker tag for master: latest
* functest-core: https://hub.docker.com/r/opnfv/functest-core/
* functest-healthcheck: https://hub.docker.com/r/opnfv/functest-healthcheck/
* functest-smoke: https://hub.docker.com/r/opnfv/functest-smoke/
- * functest-features: https://hub.docker.com/r/opnfv/functest-features/
* functest-vnf: https://hub.docker.com/r/opnfv/functest-vnf/
* functest-restapi: https://hub.docker.com/r/opnfv/functest-restapi/
If you want to create a new test case you will have to create a new folder
under the testcases directory (See next section for details).
-Functest external test cases
-============================
-The external test cases are inherited from other OPNFV projects, especially the
-feature projects.
-
-The external test cases are:
-
- * barometer
- * bgpvpn
- * doctor
- * domino
- * fds
- * promise
- * refstack_defcore
- * functest-odl-sfc
- * orchestra_clearwaterims
- * orchestra_openims
- * vyos_vrouter
- * juju_vepc
-
-External test cases integrated in previous versions but not released in
-Euphrates:
-
- * copper
- * moon
- * netready
- * security_scan
-
-
-The code to run these test cases is hosted in the repository of the project.
-Please note that orchestra test cases are hosted in Functest repository and not
-in orchestra repository. Vyos_vrouter and juju_vepc code is also hosted in
-functest as there are no dedicated projects.
-
-
Functest framework
==================
4 steps have been defined::
* functest-cleanup: clean existing functest dockers on the jumphost
- * functest-daily: run dockers opnfv/functest-* (healthcheck, smoke, features,
- vnf)
+ * functest-daily: run dockers opnfv/functest-* (healthcheck, smoke,
+ benchmarking, vnf)
* functest-store-results: push logs to artifacts
See `[1]`_ for details.
* opnfv/functest-healthcheck
* opnfv/functest-smoke
* opnfv/functest-benchmarking
- * opnfv/functest-features
* opnfv/functest-vnf
Note: if the scenario does not support some tests, they are indicated as SKIP.
See User guide for details.
-Testing features suite
-^^^^^^^^^^^^^^^^^^^^^^
-
-Run features suite::
-
- sudo docker run --env-file env \
- -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
- -v $(pwd)/images:/home/opnfv/functest/images \
- opnfv/functest-features
-
-Results shall be displayed as follows::
-
- +-----------------------------+-------------------+------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +-----------------------------+-------------------+------------------+------------------+----------------+
- | doctor-notification | doctor | features | 00:00 | SKIP |
- | functest-odl-sfc | sfc | features | 00:00 | SKIP |
- | barometercollectd | barometer | features | 00:00 | SKIP |
- | vgpu | functest | features | 00:00 | SKIP |
- | stor4nfv_os | stor4nfv | features | 00:00 | SKIP |
- +-----------------------------+-------------------+------------------+------------------+----------------+
-
-Note: if the scenario does not support some tests, they are indicated as SKIP.
-See User guide for details.
-
Testing vnf suite
^^^^^^^^^^^^^^^^^
* opnfv/functest-kubernetes-core
* opnfv/functest-kubernetest-healthcheck
* opnfv/functest-kubernetest-smoke
- * opnfv/functest-kubernetest-features
Preparing your environment
^^^^^^^^^^^^^^^^^^^^^^^^^^
| k8s_conformance | functest | smoke | 57:14 | PASS |
+-------------------------+------------------+---------------+------------------+----------------+
-Testing features suite
-^^^^^^^^^^^^^^^^^^^^^^
-
-Run features suite::
-
- sudo docker run -it --env-file env \
- -v $(pwd)/config:/root/.kube/config \
- opnfv/functest-kubernetes-features
-
-Results shall be displayed as follows::
-
- +----------------------+------------------+------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +----------------------+------------------+------------------+------------------+----------------+
- | stor4nfv_k8s | stor4nfv | stor4nfv | 00:00 | SKIP |
- | clover_k8s | clover | clover | 00:00 | SKIP |
- +----------------------+------------------+------------------+------------------+----------------+
-
Environment variables
=====================
code returned by OpenDaylight.
-Features
---------
-
-Functest has been supporting several feature projects since Brahmaputra:
-
-
-+-----------------+---------+----------+--------+-----------+-----------+
-| Test | Brahma | Colorado | Danube | Euphrates | Fraser |
-+=================+=========+==========+========+===========+===========+
-| barometer | | | X | X | X |
-+-----------------+---------+----------+--------+-----------+-----------+
-| bgpvpn | | X | X | X | X |
-+-----------------+---------+----------+--------+-----------+-----------+
-| copper | | X | | | |
-+-----------------+---------+----------+--------+-----------+-----------+
-| doctor | X | X | X | X | X |
-+-----------------+---------+----------+--------+-----------+-----------+
-| domino | | X | X | X | |
-+-----------------+---------+----------+--------+-----------+-----------+
-| fds | | | X | X | X |
-+-----------------+---------+----------+--------+-----------+-----------+
-| moon | | X | | | |
-+-----------------+---------+----------+--------+-----------+-----------+
-| multisite | | X | X | | |
-+-----------------+---------+----------+--------+-----------+-----------+
-| netready | | | X | | |
-+-----------------+---------+----------+--------+-----------+-----------+
-| odl_sfc | | X | X | X | X |
-+-----------------+---------+----------+--------+-----------+-----------+
-| opera | | | X | | |
-+-----------------+---------+----------+--------+-----------+-----------+
-| orchestra | | | X | X | X |
-+-----------------+---------+----------+--------+-----------+-----------+
-| parser | | | X | X | X |
-+-----------------+---------+----------+--------+-----------+-----------+
-| promise | X | X | X | X | X |
-+-----------------+---------+----------+--------+-----------+-----------+
-| security_scan | | X | X | | |
-+-----------------+---------+----------+--------+-----------+-----------+
-| clover | | | | | X |
-+-----------------+---------+----------+--------+-----------+-----------+
-| stor4nfv | | | | | X |
-+-----------------+---------+----------+--------+-----------+-----------+
-
-Please refer to the dedicated feature user guides for details.
-
-
VNF
---
The current list of test suites can be distributed over 5 main domains:
* VIM (Virtualised Infrastructure Manager)
* Controllers (i.e. SDN Controllers)
- * Features
* VNF (Virtual Network Functions)
* Kubernetes
Functest test suites are also distributed in the OPNFV testing categories:
-healthcheck, smoke, features, components, performance, VNF, Stress tests.
+healthcheck, smoke, benchmarking, VNF, Stress tests.
All the Healthcheck and smoke tests of a given scenario must be succesful to
validate the scenario for the release.
| | | | upstream testcases. See below |
| | | | for details |
+-------------+---------------+------------+----------------------------------+
-| Features | features | bgpvpn | Implementation of the OpenStack |
-| | | | bgpvpn API from the SDNVPN |
-| | | | feature project. It allows for |
-| | | | the creation of BGP VPNs. |
-| | | | See `SDNVPN User Guide`_ for |
-| | | | details |
-| | +------------+----------------------------------+
-| | | doctor | Doctor platform, as of Colorado |
-| | | | release, provides the three |
-| | | | features: |
-| | | | * Immediate Notification |
-| | | | * Consistent resource state |
-| | | | awareness for compute host down |
-| | | | * Valid compute host status |
-| | | | given to VM owner |
-| | | | See `Doctor User Guide`_ for |
-| | | | details |
-| | +------------+----------------------------------+
-| | | odl-sfc | SFC testing for odl scenarios |
-| | | | See `SFC User Guide`_ for details|
-| | +------------+----------------------------------+
-| | | parser | Parser is an integration project |
-| | | | which aims to provide |
-| | | | placement/deployment templates |
-| | | | translation for OPNFV platform, |
-| | | | including TOSCA -> HOT, POLICY ->|
-| | | | TOSCA and YANG -> TOSCA. it |
-| | | | deals with a fake vRNC. |
-| | | | See `Parser User Guide`_ for |
-| | | | details |
-| | +------------+----------------------------------+
-| | | fds | Test Suite for the OpenDaylight |
-| | | | SDN Controller when the GBP |
-| | | | features are installed. It |
-| | | | integrates some test suites from |
-| | | | upstream using Robot as the test |
-| | | | framework |
-+-------------+---------------+------------+----------------------------------+
| VNF | vnf | cloudify | Example of a real VNF deployment |
| | | \_ims | to show the NFV capabilities of |
| | | | the platform. The IP Multimedia |
| | smoke | k8s\_ | Run a subset of Kubernetes |
| | | conformance| End-to-End tests, expected to |
| | | | pass on any Kubernetes cluster |
-| +---------------+------------+----------------------------------+
-| | stor4nfv | stor4nfv | Run tests necessary to |
-| | | \_k8s | demonstrate conformance of the |
-| | | | K8s+Stor4NFV deployment |
-| +---------------+------------+----------------------------------+
-| | clover | clover_k8s | Test functionality of |
-| | | | K8s+Istio+Clover deployment. |
+-------------+---------------+------------+----------------------------------+
Most of the tests are runnable by any combination, but some tests might have
restrictions imposed by the utilized installers or due to the available
-deployed features. The system uses the environment variables (INSTALLER_TYPE
-and DEPLOY_SCENARIO) to automatically determine the valid test cases, for each
-given environment.
+deployed services. The system uses the environment variables to automatically
+determine the valid test cases, for each given environment.
A convenience Functest CLI utility is also available to simplify setting up the
Functest evironment, management of the OpenStack environment (e.g. resource
.. _`[2]`: https://docs.openstack.org/tempest/latest/
.. _`[3]`: https://rally.readthedocs.io/en/latest/index.html
-.. _`Doctor User Guide`: http://artifacts.opnfv.org/doctor/colorado/userguide/index.html
-.. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/colorado/docs/userguide/index.html
-.. _`Parser User Guide`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html
.. _`Functest Dashboard`: http://testresults.opnfv.org/
-.. _`SFC User Guide`: http://artifacts.opnfv.org/sfc/colorado/userguide/index.html
| neutron_trunk | functest | smoke | 00:00 | SKIP |
+------------------------------+------------------+---------------+------------------+----------------+
-Features suite::
-
- +-----------------------------+------------------------+------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +-----------------------------+------------------------+------------------+------------------+----------------+
- | doctor-notification | doctor | features | 00:00 | SKIP |
- | bgpvpn | sdnvpn | features | 00:00 | SKIP |
- | functest-odl-sfc | sfc | features | 00:00 | SKIP |
- | barometercollectd | barometer | features | 00:00 | SKIP |
- | fds | fastdatastacks | features | 00:00 | SKIP |
- +-----------------------------+------------------------+------------------+------------------+----------------+
-
Components suite::
+-------------------------------+------------------+--------------------+------------------+----------------+
| juju_epc | functest | vnf | 46:44 | PASS |
+----------------------+------------------+--------------+------------------+----------------+
-Parser testcase::
-
- +-----------------------+-----------------+------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +-----------------------+-----------------+------------------+------------------+----------------+
- | parser-basics | parser | features | 00:00 | SKIP |
- +-----------------------+-----------------+------------------+------------------+----------------+
-
Functest Kubernetes test result::
+--------------------------------------+------------------------------------------------------------+
| k8s_conformance | functest | smoke | 57:47 | PASS |
+-------------------------+------------------+---------------+------------------+----------------+
-Kubernetes features suite::
-
- +----------------------+------------------+------------------+------------------+----------------+
- | TEST CASE | PROJECT | TIER | DURATION | RESULT |
- +----------------------+------------------+------------------+------------------+----------------+
- | stor4nfv_k8s | stor4nfv | stor4nfv | 00:00 | SKIP |
- | clover_k8s | clover | clover | 00:00 | SKIP |
- +----------------------+------------------+------------------+------------------+----------------+
-
Results are automatically pushed to the test results database, some additional
result files are pushed to OPNFV artifact web sites.
run:
name: shaker
- -
- name: features
- order: 3
- description: >-
- Test suites from feature projects
- integrated in functest
- testcases:
- -
- case_name: doctor-notification
- project_name: doctor
- enabled: false
- criteria: 100
- blocking: false
- description: >-
- Test suite from Doctor project.
- dependencies:
- - INSTALLER_TYPE: '(apex)|(fuel)|(daisy)'
- - DEPLOY_SCENARIO: '^((?!fdio|nofeature).)*$'
- run:
- name: bashfeature
- args:
- cmd: 'doctor-test'
-
- -
- case_name: functest-odl-sfc
- project_name: sfc
- criteria: 100
- blocking: false
- description: >-
- Test suite for odl-sfc to test two chains with one SF and
- one chain with two SFs
- dependencies:
- - DEPLOY_SCENARIO: 'odl.*sfc'
- run:
- name: functest-odl-sfc
-
- -
- case_name: barometercollectd
- project_name: barometer
- enabled: false
- criteria: 100
- blocking: false
- description: >-
- Test suite for the Barometer project. Separate tests verify
- the proper configuration and basic functionality of all the
- collectd plugins as described in the Project Release Plan
- dependencies:
- - DEPLOY_SCENARIO: 'bar'
- run:
- name: barometercollectd
-
- -
- case_name: vgpu
- project_name: functest
- criteria: 100
- blocking: false
- description: >-
- Test suite for the OpenStack vGPU feature
- dependencies:
- - DEPLOY_SCENARIO: 'vgpu'
- run:
- name: vgpu
-
- -
- case_name: stor4nfv_os
- project_name: stor4nfv
- enabled: false
- criteria: 100
- blocking: false
- description: >-
- This tests are necessary demonstrate conformance
- of the OpenStack+Stor4NFV deployment.
- dependencies:
- - DEPLOY_SCENARIO: 'stor4nfv'
- run:
- name: stor4nfv_os
-
-
name: vnf
order: 4
Code Review / functest.git / commitdiff