Modify tests to require 100% subtest success, fix tempest 41/25941/3
authorRomanos Skiadas <rski@intracom-telecom.com>
Wed, 14 Dec 2016 08:13:23 +0000 (10:13 +0200)
committerRomanos Skiadas <rski@intracom-telecom.com>
Fri, 16 Dec 2016 12:42:45 +0000 (14:42 +0200)
commitca3cb806082b7f39ffacde3b7c7776597fa80ec9
treed2765155d53b122b2b290b9919dba0fcb9f9db64
parent89ce471643c0cbaf41cac0e6995e754b2169889b
Modify tests to require 100% subtest success, fix tempest

This is required to better align with functest, since that will handle overall
success/failure status. Also if a subestest failed, there were expected failures
that could seem like they pass making the overall success % look the same, which
would make the test look fine, even thought it failed.

Some tests check things that are known not to work. Disable these until they are
fixed upstream.

Also remove the success_criteria field from the config. Fail the test if any
subtest fails. This has the added benefit that it fixes the tempest suite, which
was still trying to read the value of `succes_criteria` that had been renamed to
`success_criteria`.

JIRA: SDNVPN-89
Change-Id: Iba5e5cc0db6c371f29a4cbba66ca16c4a1838e12
Signed-off-by: Romanos Skiadas <rski@intracom-telecom.com>
sdnvpn/lib/results.py
sdnvpn/test/functest/config.yaml
sdnvpn/test/functest/tempest.py
sdnvpn/test/functest/testcase_1.py
sdnvpn/test/functest/testcase_2.py
sdnvpn/test/functest/testcase_4.py
sdnvpn/test/functest/testcase_7.py