Change "monitor" to "assert" in SLA

Change "monitor" to "assert" in SLA to make an Overall testcase
to fail if a subtest Scenario fails

JIRA: YARDSTICK-1305

Change-Id: I90f5c95f0ddace160d751a435d60ae09a42101f2
Signed-off-by: Dimitrios Markou <mardim@intracom-telecom.com>

[Bug fix] Execute teardown if SLA is set to "assert"

In Duration runner if SLA is set to "assert" and a SLAValidationError
happens then a teardown is needed to avoid garbages in the environment.

JIRA: YARDSTICK-1291

Change-Id: I9056b0da3e7e496171ed42430ffc340b5c68f108
Signed-off-by: Dimitrios Markou <mardim@intracom-telecom.com>

Merge "Sample VNF version need to be updated to Fraser"

Merge changes from topics 'YARDSTICK-1250', 'YARDSTICK-1249'

* changes:
  Fix pip install issues in install.yaml script
  Bump ansible version to 2.5.5

Merge "Cleanup Iperf3 unit tests"

Merge "Cleanup Plugin unit tests"

Merge "Cleanup NodeContextTestCase unit tests"

Merge "Cleanup vPE VNF unit tests"

Merge "Cleanup TestIxLoadTrafficGen unit tests"

Merge "Cleanup GetDataDbClientTestCase unit tests"

Merge "Cleanup VsperfDPDKTestCase unit tests"

Sample VNF version need to be updated to Fraser

JIRA: YARDSTICK-1289

Change-Id: Ib150fafdfcc7a7aa09e4ed9c683fc3013676dac4
Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

Merge "Cleanup TestProxBinSearchProfile unit tests"

Merge "Cleanup ProxDuration unit tests"

Merge "Yardstick Fraser Release notes - 6.2.1"

Yardstick Fraser Release notes - 6.2.1

JIRA: YARDSTICK-1278

Change-Id: I96d54b06f0ef794c9c0ec26d78eb54ebcc6462ed
Signed-off-by: rexlee8776 <limingjiang@huawei.com>

Merge "Bugfix: ansible install.yaml missing configure_gui step"

Bugfix: ansible install.yaml missing configure_gui step

JIRA: YARDSTICK-1279

Change-Id: I7df97479e2e17495668ef4982de4aec13d04c25c
Signed-off-by: rexlee8776 <limingjiang@huawei.com>

Bugfix: uwsgi config file yardstick.ini output error

JIRA: YARDSTICK-1280

Change-Id: I45b4b781cee21035897ba0c34dc740af6365e918
Signed-off-by: rexlee8776 <limingjiang@huawei.com>

Bugfix: docker build failed

Due to no inventory.ini specified in Dockerfile, which
resulsts in ansible-playbook run fail

JIRA: YARDSTICK-1277

Change-Id: Iaf689b71b8c86b54b2451d8a8c480f7d94d53561
Signed-off-by: rexlee8776 <limingjiang@huawei.com>

Cleanup Iperf3 unit tests

Removed unneeded output.

JIRA: YARDSTICK-1275

Change-Id: I75fe1f7f048714ba5d2fdfa586ad69ebb7b56c88
Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

Cleanup Plugin unit tests

Removed unneeded output (LOG.info and print functions).

JIRA: YARDSTICK-1274

Change-Id: Ida3b62ce28e27058a8b4a3a9e6fa16f041a1f99c
Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

Cleanup NodeContextTestCase unit tests

Removed deprecated functions.

JIRA: YARDSTICK-1273

Change-Id: I055b8393e6b25eb08f718cd8a915c6ea684786d8
Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

Merge "Update PROX_NSB_DEVGUIDE for F Release"

Merge "Yardstick Fraser Release notes - 6.2.0"

Cleanup vPE VNF unit tests

Removed unneeded output and stopped access to file system.

JIRA: YARDSTICK-1271

Change-Id: I15a58eb7bb81b69fe156433055f59a54372fa0a8
Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

Cleanup TestIxLoadTrafficGen unit tests

Removed unneeded output and mock system calls.

JIRA: YARDSTICK-1270

Change-Id: I0e04a67cab592216280481c7145188c4f9045466
Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

Cleanup GetDataDbClientTestCase unit tests

Removed unneeded output.

JIRA: YARDSTICK-1268

Change-Id: I2e9130e863ba915feeb52f54fc901898b1c8f792
Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

Cleanup VsperfDPDKTestCase unit tests

Removed unneeded output.

JIRA: YARDSTICK-1267

Change-Id: I0a3f25db2c5ef8ed71fd087b6b35641075c121e7
Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

Cleanup TestProxBinSearchProfile unit tests

Removed unneeded output and mocks.

JIRA: YARDSTICK-1266

Change-Id: I04fd1ba32ea46c7896548faebc5af1a43b0f750e
Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

NFVI support for standalone, baremetal and heat contexts

JIRA: YARDSTICK-1257

Change-Id: I6733bd49ac91985e8f3d7722e6990e8733bb430e
Signed-off-by: Chornyi, TarasX <tarasx.chornyi@intel.com>

Update PROX_NSB_DEVGUIDE for F Release

Contains Updates :-
   New Description of Grafana output
   New Description of ProxDuration Runner
   Update Description of Openstack Network

Change-Id: I861bb7eab4aeed207c0151be6766c48c5f10c6ef
Signed-off-by: Daniel Martin Buckley <daniel.m.buckley@intel.com>

Merge "Removed unused ACL config file."

Yardstick Fraser Release notes - 6.2.0

JIRA: YARDSTICK-1261

Change-Id: I61adeaea8a727605481ffce9dae9646f160bf401
Signed-off-by: rexlee8776 <limingjiang@huawei.com>

Merge "Add testsuite "os-odl-ovs-noha""

Merge "Cleanup process test cases"

Merge "Cleanup OpenStack utils test cases"

Decrease Sampling interval - Invalid Port results

JIRA: YARDSTICK-1219

The computed in/fwd throughputs displayed on the L2FWD,
L3FWD and VPE dashboards. The computed value is likely
4x the expected value.

It is required to return sampling information form Generator AND VNF at
least every  1 second.

Change-Id: I4435fd05ba3116ead836843a4c2fce133b767a28
Signed-off-by: Daniel Martin Buckley <daniel.m.buckley@intel.com>

Cleanup ProxDuration unit tests

Removed unneeded output, removed access to files, fixed indentation,
bumped deprecated asserts, reduced time execution and fixed file header.

JIRA: YARDSTICK-1262

Change-Id: I4803980748d4731cc27608e386707a858e1a8630
Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

Removed unused ACL config file.

Change-Id: Ibdb1dd12587df0182b5e8d3b7bc6901be0aac3ec
Signed-off-by: Mytnyk, Volodymyr <volodymyrx.mytnyk@intel.com>

Merge "Update to Grafana Dashboard for NSB PROX NFVI"

Merge "NSB NFVi PROX BNG losing many packets"

NSB NFVi PROX BNG losing many packets

JIRA: YARDSTICK-1101

Problem was incorrect generator and changed the traffic profile to allow
a tolerated loss of 5%.

ARP packets are not managed (just passed thru the system) therefor the
tolerated loss must be higehr than normal.

Change-Id: I03bd62b397a66b50961e810d033c6894d0b3e9a6
Signed-off-by: Daniel Martin Buckley <daniel.m.buckley@intel.com>

Update to Grafana Dashboard for NSB PROX NFVI

JIRA: YARDSTICK-1252

CPU grouping in grafana for better representation
Throughput table is difficult to read as the tests
  run longer the values can overlap
Default time in dashboard should be always last 1hour in
  dashboard (auto refresh can be also checked)
Add one column in table with Gbps throughput (Rx
  Throughput) plus code change in yardstick
Boxes are not aligned and big and small, need to be fixed
  boxes are now aligned,
Added Seperate graphe for LINE Rate

Change-Id: Ief31777c2e7a18aab4bf21fb6ccc0e93a3bb9178
Signed-off-by: Daniel Martin Buckley <daniel.m.buckley@intel.com>

Merge "Fix NSB NfVi support for 25 and 40Gbps"

Corrected scale-up command line arguments

Change-Id: If308fde797dcb01e58f37ad81bb116fde542366a
Signed-off-by: Chornyi, TarasX <tarasx.chornyi@intel.com>

Fix NSB NfVi support for 25 and 40Gbps

When setting interface_speed option in the test case, NSB/PROX
was still considering 100% as 10 Gbps.
With this fix, when interface_speed is set to for instance 25 Gbps
then NSB/PROX will start generate at 25 Gbps.
Yardstick logging info printed on the screen will show performance result
as percentage of interface_speed (e.g. 100% will be 25 Gbps).
This is done by sending the command "speed 250" to PROX (PROX cannot
know the underlying interface speed in all cases e.g. in virtio, and
hence considers 100% as 10Gbps).

JIRA: YARDSTICK-1217

Change-Id: I4989161b04a7afa5e347935ee7ccef7dfe5a504d
Signed-off-by: Xavier Simonart <xavier.simonart@intel.com>
Signed-off-by: Emma Foley <emma.l.foley@intel.com>

Merge "Resolve NameError in test_utils.py"

NSB NFVi PROX VPE losing many packets

JIRA: YARDSTICK-1105

Running a vPE test ends up with up to 25% drooped packets.
This is due to tolerated loss being incorrectly configured,
test duration was too short and test interval was too
short.

Note This is due to the fact that vPE use case have by
default a tolerated loss of 100%, Now changed to 0.001,
same as L2FWD

Also: Grafan dashboard has been updated to show all SUT
CPUS Utilization in 1 panel. And LINE rate MIN, MAX & TEST
Rate have been added

Change-Id: I7adae2199b3f656fe460705b6aeb3aa69c767d50
Signed-off-by: Daniel Martin Buckley <daniel.m.buckley@intel.com>

Add testsuite "os-odl-ovs-noha"

Add testsuite yaml file for scenario
"os-odl-ovs-noha"

JIRA: YARDSTICK-1208

Change-Id: I2671df847332379afed8c6481e54eb25dcba5e48
Signed-off-by: ting wu <ting.wu@enea.com>

Merge "Define several containers per pod in a replication controller"

Merge "Decrease Sampling interval"

NSB NFVi BNG test fails to run - stops after one step

JIRA: YARDSTICK-1244

When sending the start or stop command to PROX. We sould not send the same core.
We should not send "start 1,1" instead we should send "start 1"

Change-Id: Ie600dc3ed808fd00341e92f13bd52199e68dc51f
Signed-off-by: Daniel Martin Buckley <daniel.m.buckley@intel.com>

Decrease Sampling interval

JIRA: YARDSTICK-1219

Currently NSB for NFVI using PROX returns sampling information every 13
seconds.

This is too slow.

It is required to return sampling information form Generator AND VNF at
least every  1 second.

This change is depandant on JIRA: YARDSTICK-1212 and YARDSTICK-1220

Change-Id: Ica7ab795a2919d191d8cd846d028e15739e33fb7
Signed-off-by: Daniel Martin Buckley <daniel.m.buckley@intel.com>

Merge "Addition of Configurable Sampling and Configurable Confirmation Retry"

Merge "Bugfix: heat conext of test case yamls jinja2 bypass sriov type"

Resolve NameError in test_utils.py

The import of the yardstick module was removed from test_util.py, which relied
on this module location for a file name in
test_import_modules_from_package_no_mod and test_import_modules_from_package.

JIRA: YARDSTICK-1258
Change-Id: I754dbadfb8d788236f354e9a0c1a61ad6c4abf70
Signed-off-by: Emma Foley <emma.l.foley@intel.com>

Merge "Extended Context class with get_physical_nodes functionality"

Addition of Configurable Sampling and Configurable Confirmation Retry

JIRA: YARDSTICK-1221

In order to increase accuracy of result the following are done :-
  - Improve Measurement Accuracy (YARDSTICK-1212)
  - Improve Sampling Interval (YARDSTICK-1219)
  - Allow 4 PROX ports to be read simultaneously (YARDSTICK-1220)

This change does the following :-
  - Stores LINE Rate statistics of Sample
  - Requires a confirmation retry before deciding to increase or
    decrease this is configurable.
  - Allows the user to disable Sampling or specify a sample interval
  - Added Code Coverage of ProxDurationRunner based on YARDSTICK-1199

Change-Id: I27242ac1849c9a2712866385b5fbc05977c71516
Signed-off-by: Daniel Martin Buckley <daniel.m.buckley@intel.com>

Merge "Bugfix: HA kill process recovery has a conflict"

Merge "NSB Topology fix for Prox 4 port test case"

Merge "Refactor "utils.parse_ini_file" testing"

Merge "Refactor remote command execution in pktgen_dpdk"

Extended Context class with get_physical_nodes functionality

JIRA: YARDSTICK-1255

Change-Id: I446d715dc6cf716a4fcbc1b809c1b1d6303071e0
Signed-off-by: Chornyi, TarasX <tarasx.chornyi@intel.com>

Merge "Add unit test file for DurationRunner"

Merge "Refactor remote command execution in pktgen"

NSB Topology fix for Prox 4 port test case

The uplink and downlink interfaces for the 4 port
test cases were inconsistent and fixed in this commit.

This affects all the 4 port PROX test cases.

Tested on: Heat L2fwd-4 and L3fwd-4 test case.

JIRA: YARDSTICK-1185

Change-Id: Ia2ce7ed0dc395b812f84ede94e259b42dc833579
Signed-off-by:Abhijit Sinha <abhijit.sinha@intel.com>

Bugfix: HA kill process recovery has a conflict

It happens in Nokia SUT when running in the Plugfest. The problem happens when
the start_process start to recover the killed process (like nova-api), but the
self-cured mechanism already recovered. And somehow it lead to a conflict and
has problems.
So the recover of the HA attack-recover should be improved to only recover it
when it needs to.

JIRA: YARDSTICK-1222

Change-Id: I1acb5a7d59d6fe4e0de0b0c5942fa89e051dd1ff
Signed-off-by: rexlee8776 <limingjiang@huawei.com>

Add unit test file for DurationRunner

JIRA: YARDSTICK-1199

Change-Id: I5526935207ae3661a4cb8b8d25e93bc26adc3ef2
Signed-off-by: Miikka Koistinen <miikka.koistinen@nokia.com>

Merge "Cleanup CGNAPT unit tests"

Merge "Add "volumeMounts" parameter in Kubernetes context"

Merge "Add "volumes" parameter in Kubernetes context"

Change IxNextgen library load

Changed IxNextgen library load in IXIA RFC2544 traffic generator call.

JIRA: YARDSTICK-1116

Change-Id: Icbdc5f1255dd2676584f3a1be767287e55e71f4a
Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

Merge "Configure ACL via static file"

Configure ACL via static file

This patch allows user to configure ACL/vFW SampleVNF ACL
via configuration file provided in TC definition. The
Yardstick applies the rules to SampleVNF from specified config
file + rules generated by Yardstick (default rules).

The example of SampleVNF ACL CLI commands generated/applied
by Yardstick can be found at (using default ACL config file):

ACL VNF ACL CLI commands:
  acl/tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex.yaml
  http://paste.openstack.org/show/723303/

vFW VNF ACL CLI commands:
  vfw/tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex.yaml
  http://paste.openstack.org/show/723304/

Change-Id: I76a630261a982083b628e3985fc3bec14ca495db
Signed-off-by: Mytnyk, Volodymyr <volodymyrx.mytnyk@intel.com>

Merge "Cleanup EnvCommand test cases"

Merge "Add scale out TCs with availability zone support"

Fix pip install issues in install.yaml script

For long executing commands, ansible should use async jobs. For example,
when installing the project requirements.

There is a problem in ansible pip module when using a "editable installs".
This command should be called with "editable" flag.

JIRA: YARDSTICK-1250

Change-Id: I6d29bab2f388373e252d065ff5aa0c459a8f265a
Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

Bump ansible version to 2.5.5

Increase ansible version to 2.5.5.

There is a closed bug present in version 2.4.2 which affects to some ansible
scripts. When calling recursively different roles, the maximum recursion
depth can be exceeded [1]. This bug is fixed in proposed version.

[1] https://github.com/ansible/ansible/issues/23609

JIRA: YARDSTICK-1249

Change-Id: Ia45b6a2ab44203e39c52357eb3cfee1e0c77eed3
Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

Merge "Document for Fraser test case results"

NSB setup inventory name changed

NSB setup script [1] fails when retrieving the inventory file. This script
is going to be deprecated but we still support it.

[1] https://github.com/opnfv/yardstick/blob/master/nsb_setup.sh

JIRA: YARDSTICK-1247

Change-Id: I3798531652b0e3a1a007d1d9434998fdd54645d7
Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

Merge "Refactor RFC2455 TRex traffic profile injection"

Merge "Add "duration" parameter to test case definition"

Merge "Update pmd/lcore mask for OVS-DPDK context"

Refactor remote command execution in pktgen_dpdk

PktgenDPDKLatency.run will raise a RuntimeError if a remote command
fails. This commit makes it use the already existing exception raising
mechanism in yardstick's ssh client.

JIRA: YARDSTICK-1166

Change-Id: I3a3c7691399044b174f5d040c015c9b907b2fe5d
Signed-off-by: Miikka Koistinen <miikka.koistinen@nokia.com>

Refactor remote command execution in pktgen

Some remote commands in Pktgen are executed without checking the exit
status. This patch makes all remote commands check exit value, and
removes unused variables that are captured from remote command
executions.

JIRA: YARDSTICK-1166

Change-Id: I42a667ebd22d086887d61e1671bc569b03c59d33
Signed-off-by: Miikka Koistinen <miikka.koistinen@nokia.com>

Update pmd/lcore mask for OVS-DPDK context

Added default values for pmd/lcore mask to the SampleVNF test case for
Trex TG.
lcore mask - specifies the CPU cores used for non-datapath OVS-DPDK
threads.
pmd mask - specifies the CPU cores used for data path packet processing.
Updated documentation for Standalone OVS-DPDK context.

JIRA: YARDSTICK-1246

Change-Id: I8d008a185c94d2047ffa4769d7974bc4d283fd7f
Signed-off-by: Stepan Andrushko <stepanx.andrushko@intel.com>

Cleanup EnvCommand test cases

Refactor exception testing and remove console output.

JIRA: YARDSTICK-1235

Change-Id: I6376c3637ffb6476a2578ef963cd2420b63bab52
Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

Refactor RFC2455 TRex traffic profile injection

The TRex RFC2455 traffic profile configures the duration of the traffic
injection. Once the traffic is started, the traffic profile should poll
the traffic injector client for the injection status. When the injection
is finished, the traffic execution should end.

Instead of this, the traffic profile waits a fixed time (using time.sleep
method) and then stops the traffic. This approach is not accurate and may
cause shorter injection periods.

JIRA: YARDSTICK-1197

Change-Id: I5090df81ef4ec7945ff6c1aff070656b48e0fe77
Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

Add "duration" parameter to test case definition

Add "duration" parameter to test case definition, in scenario:options
section. This parameter will be rendered in the traffic profile.

If the parameter is not present in the test case scenario options, the
default time written in the traffic profile options will be 30 seconds
(TrafficProfile.DEFAULT_DURATION = 30). If the traffic profile
injection time is not defined, the default injection time will be 30
seconds.

  testcase:scenario:options:duration (default = 30)
      render --> traffic_profile:duration
          parse --> TrafficProfile.duration (default = 30)

Target traffic profiles (RFC2544):
  - RFC2544Profile
  - IXIARFC2544Profile

JIRA: YARDSTICK-1194

Change-Id: I968922e6bb882d7ee15aa1c4db4037face7a3492
Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

Merge changes from topic 'convert-sla-asserts-to-raises'

* changes:
  Replace MagicMocks with Mocks from SearchRunner UTs
  Add vsperf_dpdk SLA validation unit test cases
  Cleanup vsperf unit tests

Merge "Convert SLA asserts to raises"

Stop running tests from tests/unit

Since all tests have been moved into yardstick/tests,
temperary code to run tests in both tests and yardstick/tests
should be removed.

JIRA: YARDSTICK-837
Change-Id: I3bf3c9ece9bc8457d992e53ee8cc5cadb1dec8f0
Signed-off-by: Emma Foley <emma.l.foley@intel.com>

Merge "Cleanup BaseMonitor unit tests"

Cleanup BaseMonitor unit tests

Stop multiprocessing queues to avoid BrokenPipe exception.
Assert correctly the raise of exceptions.

JIRA: YARDSTICK-1229

Change-Id: I1a1c92c49a439283a35c9a69b59d6866f35ec09a
Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

Replace MagicMocks with Mocks from SearchRunner UTs

This patch replaces unneeded MagicMocks with Mocks from TestSearchRunner
and TestSearchRunnerHelper.

JIRA: YARDSTICK-1169

Change-Id: I30a7bbc71c0b1062e6c0dfe634a5a8f1ecb7c684
Signed-off-by: Miikka Koistinen <miikka.koistinen@nokia.com>

Convert SLA asserts to raises

This commit converts Python assertions to a custom exception in all
places where SLA validation is checked with an assertion.

This commit also fixes all emerged pylint errors.

JIRA: YARDSTICK-966

Change-Id: If771ed03b2cbc0a43a57fcfb9293f18740b3ff80
Signed-off-by: Miikka Koistinen <miikka.koistinen@nokia.com>

Add vsperf_dpdk SLA validation unit test cases

JIRA: YARDSTICK-1228

Change-Id: Ic32f2bcd7f8bdf718c9c266666409d32ecab4924
Signed-off-by: Miikka Koistinen <miikka.koistinen@nokia.com>

Cleanup vsperf unit tests

Remove unneeded mock return values and improve the test execution.

Add unit test cases about SLA validation.

JIRA: YARDSTICK-1227

Change-Id: I90b8b9c4cb6a57ae4b0869e84af0da43ee22a670
Signed-off-by: Miikka Koistinen <miikka.koistinen@nokia.com>

Merge "assert[Greater,Equal] -> assert_{,not_}called"