Add a new testcase: cnf_testsuite_workload

It runs all the workload test plan vs the example coredns.
Please note criteria is arbitrary set to 50%. Testing vs coredns
is about 80% and envoy 70%.

It also add extra processing to calculate results in
the main 3 cases (cert, workload and single tag).

Change-Id: I5e161727cab40228991ab97a061b13b335f87b4c
Signed-off-by: Cédric Ollivier <cedric.ollivier@orange.com>
(cherry picked from commit b716ddbfd7efe66cf9fbe73838d9eec2e3350025)

diff --git a/docker/cnf/testcases.yaml b/docker/cnf/testcases.yaml
index be28963..704f2e9 100644 (file)
--- a/docker/cnf/testcases.yaml
+++ b/docker/cnf/testcases.yaml
@@ -30,6 +30,9 @@ tiers:
 
       - case_name: cnf_testsuite
         project_name: functest
+        # https://github.com/lfn-cnti/certification/blob/main/docs/CNTiCertification-2.0-beta.md
+        # V2.0-beta certification requires passing at least 15 of
+        # the 19 total Essential tests.
         criteria: 15
         blocking: false
         description: >-
@@ -43,3 +46,18 @@ tiers:
           args:
             cnf-config: example-cnfs/coredns/cnf-testsuite.yml
             tag: cert
+      - case_name: cnf_testsuite_workload
+        project_name: functest
+        criteria: 50
+        blocking: false
+        description: >-
+          Enable interoperability of Cloud native Network
+          Functions (CNFs) as proposed by
+          https://github.com/cnti-testcatalog/testsuite
+        dependencies:
+          - DEPLOY_SCENARIO: "k8-*"
+        run:
+          name: cnf_testsuite
+          args:
+            cnf-config: example-cnfs/coredns/cnf-testsuite.yml
+            tag: workload

diff --git a/functest_kubernetes/cnf_conformance/conformance.py b/functest_kubernetes/cnf_conformance/conformance.py
index 69069ca..5b29ad8 100644 (file)
--- a/functest_kubernetes/cnf_conformance/conformance.py
+++ b/functest_kubernetes/cnf_conformance/conformance.py
@@ -136,13 +136,21 @@ class CNFConformance(testcase.TestCase):
                     self.__logger.warning(
                         "%s %s", item['name'], item['status'])
             self.__logger.info("\n\n%s\n", msg.get_string())
-        grp = re.search(
-            r'(\d+) of (\d+) essential tests passed', output.decode("utf-8"))
-        if grp:
-            # https://github.com/cncf/cnf-certification/blob/main/reviewing.md
-            self.result = int(grp.group(1))
+        if kwargs.get("tag", self.default_tag) == 'cert':
+            grp = re.search(
+                r'(\d+) of (\d+) essential tests passed',
+                output.decode("utf-8"))
+            if grp:
+                self.result = int(grp.group(1))
+            else:
+                self.result = 0
         else:
-            self.result = 0
+            grp = re.search(
+                r'Final .* score: (\d+) of (\d+)', output.decode("utf-8"))
+            if grp:
+                self.result = int(grp.group(1)) / int(grp.group(2)) * 100
+            else:
+                self.result = item_criteria / len(self.details['items']) * 100
         if not os.path.exists(self.res_dir):
             os.makedirs(self.res_dir)
         shutil.copy2(

diff --git a/tox.ini b/tox.ini
index 3d35d70..4612ce8 100644 (file)
--- a/tox.ini
+++ b/tox.ini
@@ -5,7 +5,7 @@ envlist = pep8,pylint,yamllint,bashate,py310,bandit,perm
 usedevelop = True
 deps =
   -chttps://git.opnfv.org/functest/plain/upper-constraints.txt?h=stable/zed
-  -c{env:UPPER_CONSTRAINTS_FILE:https://opendev.org/openstack/requirements/raw/branch/stable/zed/upper-constraints.txt}
+  -c{env:UPPER_CONSTRAINTS_FILE:https://opendev.org/openstack/requirements/raw/branch/unmaintained/zed/upper-constraints.txt}
   -r{toxinidir}/requirements.txt
   -r{toxinidir}/test-requirements.txt
 install_command = pip install --use-deprecated=legacy-resolver {opts} {packages}