storperf.git
8 years agoRemove deprecated fields 75/29875/1
mbeierl [Mon, 6 Mar 2017 19:53:40 +0000 (14:53 -0500)]
Remove deprecated fields

Removes the nowarm and nossd fields as it is better to be
explicit about running the '_warm_up' workload.

Change-Id: I54b67006abdc071f6fd190d6a5eefe026609c150
JIRA: STORPERF-108
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoFixing final report 69/29369/2
mbeierl [Fri, 24 Feb 2017 19:29:08 +0000 (14:29 -0500)]
Fixing final report

Makes the report readable instead of JSON escaped and dumps it
to the Jenkins daily when done.

Change-Id: Ic3788672b185461369e2c41221ae36fc4b45436b
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoDocumentation Update 67/29167/8
mbeierl [Tue, 21 Feb 2017 18:23:42 +0000 (13:23 -0500)]
Documentation Update

Adds documentation for Danube.

Change-Id: I2755cb00b769cd0668e5c956d0b70c33d328d882
JIRA: STORPERF-81
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoSteady State Metrics 97/28997/1
mbeierl [Fri, 17 Feb 2017 22:43:39 +0000 (17:43 -0500)]
Steady State Metrics

Changes the overall value of all metrics to be based on the
calculated steady state values instead of the average for the
entire run.

Change-Id: I121929d5fe2dd43df7f289b82e9f5291c9ea9aab
JIRA: STORPERF-107
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoPrevent notifications after termination 55/28855/1
mbeierl [Thu, 16 Feb 2017 15:40:35 +0000 (10:40 -0500)]
Prevent notifications after termination

Changes the event notification logic inside the FIO invoker
so that it no longer publishes events after termination.
Prevents false reports after steady state has been detected.

Change-Id: I694f77b6493b88820fe4f4cc7f634e3e62c45a9a
JIRA: STORPERF-105
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoNormalize data series 57/28757/1
mbeierl [Wed, 15 Feb 2017 21:38:29 +0000 (16:38 -0500)]
Normalize data series

Changes the data series from using seconds to samples
so that slope is not artificially flattened.

Change-Id: Idf87926a47c2ba67e66e2254d3572adad7a81b44
JIRA: STORPERF-106
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoAdd URL for results report 65/27865/1
mbeierl [Wed, 1 Feb 2017 19:27:19 +0000 (14:27 -0500)]
Add URL for results report
Records the URL returned from testresults db.

Changes the URL ref from localhost to external and reports in
daily job.

Change-Id: I0068ea963671fb183779ac20592ba6933647eea0
JIRA: STORPERF-104
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoFixing double JSON encoding of results payload 11/27811/1
mbeierl [Tue, 31 Jan 2017 17:57:56 +0000 (12:57 -0500)]
Fixing double JSON encoding of results payload

Change-Id: I5e5a8180d5851630b1200ade84f6beb5dfa18466
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoMake it easier to see what's happening in Jenkins 59/27759/1
mbeierl [Mon, 30 Jan 2017 17:14:02 +0000 (12:14 -0500)]
Make it easier to see what's happening in Jenkins

Change-Id: I3ac727faa98c1e87dcd9522c0063ca6b251c62b4
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoTestresults DB Alignment 85/27685/1
mbeierl [Fri, 27 Jan 2017 21:35:37 +0000 (16:35 -0500)]
Testresults DB Alignment

Changes the key names sent to the testresults db to align with other
projects.

Defines a clear PASS/FAIL based on the ability to find steady state for
all runs.  If one run failed to achieve steady state, run gets labelled
as FAIL.

Change-Id: I73143779453689eea93829079388ed8035134aa3
JIRA: STORPERF-100
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoDump env so I can see what variables are visible once the job is run under Jenkins 83/27683/2
mbeierl [Fri, 27 Jan 2017 18:15:53 +0000 (13:15 -0500)]
Dump env so I can see what variables are visible once the job is run under Jenkins

Change-Id: I576a75864fa0d7c6d12990c91984e12b1489258f
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoFixed line to long error 81/27681/1
mbeierl [Fri, 27 Jan 2017 17:44:07 +0000 (12:44 -0500)]
Fixed line to long error

Fixes the version of requests to match what docker was already
pulling in.

Truncate log message to 512 characters.

Change-Id: Idba9374c0f339c8b4cb6902c921c8b26fef573cc
JIRA: STORPERF-103
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoPin version of setuptools 27/27627/1
mbeierl [Thu, 26 Jan 2017 18:00:16 +0000 (13:00 -0500)]
Pin version of setuptools

Forces explicit version of setuptools to be used because
of an upstream change:
https://github.com/pypa/setuptools/issues/937

Change-Id: I4124dedee38713ca1bfa445944d8d79cf1be6c15
JIRA: STORPERF-102
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoHardening FIO interaction 25/27625/3
mbeierl [Thu, 26 Jan 2017 16:30:26 +0000 (11:30 -0500)]
Hardening FIO interaction

Fixes a problem where FIO does not terminate by scheduling a
second killall if we get a specific message back from FIO
stderr.

Introduces a new flavor for StorPerf that has a little more
memory as larger memory maps for duplicate blocks sometimes
caused out of memory killer to be invoked.

Change-Id: I06856561ad73fef582a81d4136a36a1bea47654a
JIRA: STORPERF-99
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoFix multiple workload runs 19/27319/1
mbeierl [Fri, 20 Jan 2017 21:05:24 +0000 (16:05 -0500)]
Fix multiple workload runs

Change reporting so that multiple workloads in one job execution can
be reported instead of overwriting the previous value.

Change the daily job to use a single, multiple workload run.

Change-Id: I8e350350ae13d2272b584af7a60ad269de160587
JIRA: STORPERF-98
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoCorrect duplicate event notification 11/27311/6
mbeierl [Fri, 20 Jan 2017 17:25:19 +0000 (12:25 -0500)]
Correct duplicate event notification

Clears the listener list on start and remove listener after run
is completed.

Change-Id: I76e1f13379d60bdb72fcf2faf57d9762a62f17d3
JIRA: STORPERF-96
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoChange to Ubuntu 16.04 07/27307/1
mbeierl [Fri, 20 Jan 2017 16:40:06 +0000 (11:40 -0500)]
Change to Ubuntu 16.04

Change the base image used for the agent VM to Ubuntu 16.04 as
the 14.04 image has kernel panic when the Cinder Volume is detached

Change-Id: I37e41939826fbc7c1efcc4360b99b1a2b34b8d11
JIRA: STORPERF-97
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoActually check out releng locally 89/27189/1
mbeierl [Wed, 18 Jan 2017 20:09:34 +0000 (15:09 -0500)]
Actually check out releng locally

Change-Id: I50358e0685be45abd9330639fcdfa9c8a39ff5ba
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoFixing up paths and checking out releng locally 87/27187/2
mbeierl [Wed, 18 Jan 2017 19:22:08 +0000 (14:22 -0500)]
Fixing up paths and checking out releng locally

Change-Id: I62d877acc37a10391e59db1611c0a0af3d57fca3
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoFixed typo 85/27185/1
mbeierl [Wed, 18 Jan 2017 19:07:11 +0000 (14:07 -0500)]
Fixed typo

Change-Id: Iedc8f023636325dee664bfafab45381997d6403a
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoStorPerf Daily Job 23/27123/2
mbeierl [Tue, 17 Jan 2017 19:22:18 +0000 (14:22 -0500)]
StorPerf Daily Job

Helper script that determines what installer was used.
Script to generate RC file for container.
Script to determine number of Cinder nodes.
Daily job to pull new container, create stack based on
number of Cinder nodes, issue a warm up, and then run
a series of performance tests against the stack.  Deletes
stack once complete.

JIRA: STORPERF-87

Signed-off-by: mbeierl <mark.beierl@dell.com>
Change-Id: I551f21ffc85f11b0c0724205e952551022a94f5b

8 years agoExpose TENANT_ID 19/27119/1
mbeierl [Tue, 17 Jan 2017 19:16:29 +0000 (14:16 -0500)]
Expose TENANT_ID

Change-Id: I2b07bb9724b19f8b25e35160fba3bf11621a30d7
JIRA: STORPERF-93
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoUse v3 Auth 49/25749/4
mbeierl [Fri, 9 Dec 2016 18:34:35 +0000 (13:34 -0500)]
Use v3 Auth

Change to detect environment variable and use v3 authentication
instead if present.

Change-Id: Iada49741b3297f20b740e232b2b35bec92fda91a
JIRA: STORPERF-88
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoAllow 0 to match steady state and don't terminate the warmup 47/25547/1
mbeierl [Tue, 6 Dec 2016 15:54:10 +0000 (10:54 -0500)]
Allow 0 to match steady state and don't terminate the warmup

Change-Id: I61f8ce4e719e2c2ef7597c0e41aff8c8d79edc2d
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoFixed typo 43/25543/1
mbeierl [Tue, 6 Dec 2016 14:45:04 +0000 (09:45 -0500)]
Fixed typo

Change-Id: I82cf026017690436a1112daabfa0b084275d8fe7
Signed-off-by: mbeierl <mark.beierl@dell.com>
8 years agoSteady state detection 45/25445/3
Mark Beierl [Sat, 3 Dec 2016 03:25:48 +0000 (22:25 -0500)]
Steady state detection

Detection of steady state after 10+ samples of data

Change-Id: I29368b06188c6370d17b3d567fece6486d171235
JIRA: STORPERF-72 STORPERF-73
Signed-off-by: Mark Beierl <mark.beierl@dell.com>
8 years agoMerge "Change to Paramiko"
Mark Beierl [Sat, 3 Dec 2016 03:24:07 +0000 (03:24 +0000)]
Merge "Change to Paramiko"

8 years agoChange to Paramiko 33/25433/2
Mark Beierl [Fri, 2 Dec 2016 16:19:54 +0000 (11:19 -0500)]
Change to Paramiko

Use Paramiko ssh client instead of invoking ssh and scp
from the command line

Change-Id: Ibc8395b98842fd7f40b49c4dafa2688d8e64abc7
JIRA: STORPERF-91
Signed-off-by: Mark Beierl <mark.beierl@dell.com>
8 years agobugfix: Add BRANCH Argument in the Dockerfile 95/25295/1
MatthewLi [Thu, 1 Dec 2016 03:34:47 +0000 (22:34 -0500)]
bugfix: Add BRANCH Argument in the Dockerfile

Change-Id: If5ccbe0633ed5331b38d8231e0dd6eadd05ecbe8
Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
8 years agoData Handling Refactoring 43/25043/3
Mark Beierl [Mon, 28 Nov 2016 21:17:05 +0000 (16:17 -0500)]
Data Handling Refactoring

Break out test db interaction into new module and
make the push event driven instead of the sleep
that was there before.

Change-Id: I9485aba1405f6c3b4ee5000168fbc037efa87c81
JIRA: STORPERF-90
Signed-off-by: Mark Beierl <mark.beierl@dell.com>
8 years agoAdd BRANCH Argument in the Dockerfile 51/25251/1
jose.lausuch [Wed, 30 Nov 2016 14:49:47 +0000 (15:49 +0100)]
Add BRANCH Argument in the Dockerfile

This patch allows:
   - clone the respective repo branch while building the image
   - slightly reduce the size of the image by cloning only the latest

Change-Id: I48994d7d9cf230493fa22f39cf1ed7ef4fd72d9e
Signed-off-by: jose.lausuch <jose.lausuch@ericsson.com>
8 years agoData Reporting Gate 11/24911/2
Mark Beierl [Fri, 25 Nov 2016 18:28:40 +0000 (13:28 -0500)]
Data Reporting Gate

Module that allows passing of a gate only once all other
peers have also reported in within a specified time period

Change-Id: If4baf1d4377026c7833a6f30bfc2e36698f675e8
JIRA: STORPERF-71
Signed-off-by: Mark Beierl <mark.beierl@dell.com>
8 years agoDaily Job 87/24787/5
Mark Beierl [Wed, 23 Nov 2016 21:59:18 +0000 (16:59 -0500)]
Daily Job

Creation of a daily job script that runs under Jenkins and
kicks off a series of tests using the freshly cloned workspace.

Change-Id: Ibc63c1df954578ad78604321ea410c8fd8c63c41
JIRA: STORPERF-87
Signed-off-by: Mark Beierl <mark.beierl@dell.com>
8 years agoAbility to query job status 93/24693/4
Mark Beierl [Tue, 22 Nov 2016 20:45:17 +0000 (15:45 -0500)]
Ability to query job status

Provide a method to query the status of a job by id

Change-Id: I4d3567995e7c7c3359596346009358276e79d65c
JIRA: STORPERF-89
Signed-off-by: Mark Beierl <mark.beierl@dell.com>
8 years agoWait for agent status 71/24671/1
Mark Beierl [Tue, 22 Nov 2016 15:26:06 +0000 (10:26 -0500)]
Wait for agent status

Test the agent's ssh port prior to proceeding with test

Change-Id: Iff65f2c12a50063aa9f2ee4b75638ec0666ded1b
JIRA: STORPERF-34
Signed-off-by: Mark Beierl <mark.beierl@dell.com>
8 years agoAdditional Coverage 41/24241/1
Mark Beierl [Fri, 11 Nov 2016 15:48:38 +0000 (10:48 -0500)]
Additional Coverage

Adds HTML output to coverage so we can see it in Jenkins

Change-Id: I89595921bf86b8963890051a25e45f7bd08b0636
Signed-off-by: Mark Beierl <mark.beierl@dell.com>
8 years agoCreate Stack Fixes 97/24197/2
Mark Beierl [Thu, 10 Nov 2016 20:47:41 +0000 (15:47 -0500)]
Create Stack Fixes

Removes check stack as that can timeout when a large
number of agents are requested

Waits for stack to be created before returning

JIRA: STORPERF-70

Change-Id: I1cbe45b1f60f0ecab0691f0822ed5643778d218b
Signed-off-by: Mark Beierl <mark.beierl@dell.com>
8 years agoInstallation Guide 31/20331/3
mbeierl [Mon, 5 Sep 2016 12:13:28 +0000 (08:13 -0400)]
Installation Guide

Populate the installation guide

Change-Id: I271ee6d5a38195aaf95a9c821bee8049470d674f
JIRA: DOCS-124
Signed-off-by: mbeierl <mark.beierl@emc.com>
8 years agoSlope error 29/19529/2
Mark Beierl [Wed, 24 Aug 2016 19:30:15 +0000 (15:30 -0400)]
Slope error

Test and fix for error in slope
STORPERF-74

Change-Id: I17a8e0e1e588deb875ecace85290f92167df1d5e
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
8 years agoAdd Steady State Detection module 97/16997/2
Tim Rault [Fri, 15 Jul 2016 20:32:51 +0000 (16:32 -0400)]
Add Steady State Detection module

Added a Steady State Detection module containing a steady_state(data_series)
function that is able to return a boolean indicating wether or not steady
state is reached with the data_series being passed.
This module requires a data_treatment(data_series) and an average(data_series)
modules that have been added in this commit as well. The data treatment function
aims at formatting the data series that is passed to the high level steady_state
function to reach the requirement of each sub-module (slope, average and range).

Modified the Slope and Range functions so they return None when passed an empty
data series instead of 0 which was wrong. Modified the corresponding test cases.

Modified the math_range_test.py file to fix a bug in the 2 last tests.

Change-Id: I9c3854cb0a21cc37b0787b8afca0821eefaa203d
JIRA: STORPERF-60
JIRA: STORPERF-59
JIRA: STORPERF-61
JIRA: STORPERF-62
Signed-off-by: Tim Rault <tim.rault@cengn.ca>
8 years agoSeparation of test and source 25/16925/1
Mark Beierl [Thu, 14 Jul 2016 20:20:44 +0000 (16:20 -0400)]
Separation of test and source

Moving the test files into their own top-level
directory to keep things clean

Change-Id: Ic50b881045bc59b003807923424345b335dd5c95
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
8 years agoEnforce Style Guidelines 73/16873/2
Mark Beierl [Thu, 14 Jul 2016 03:26:00 +0000 (23:26 -0400)]
Enforce Style Guidelines

Update to fail verify if pep8 compliance not met.
Revert back to Coverage 4.1 to show code coverage
report in Jenkins again

Change-Id: Icb467451f101942255c7944ad796497609dcb07e
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
8 years agoMerge "Fix missing test"
Mark Beierl [Thu, 14 Jul 2016 20:08:51 +0000 (20:08 +0000)]
Merge "Fix missing test"

8 years agoAdd Range function for Steady State detection 81/16781/2
Tim Rault [Tue, 12 Jul 2016 19:30:51 +0000 (15:30 -0400)]
Add Range function for Steady State detection

Added a range_value function in utilities/math.py able to compute the range
of a series of y values : [y1, y2, ..., yn].
Implemented a test harness for this range_value function in the tests/utilities
section.
Renamed the math_slope.py and math_range.py test files to add _test.py for
Jenkins.
Cleaned up the code so it is compliant to the pep8 rules.

Renamed the previous 'math' modules (storperf/utilities/math.py
and storperf/test/utilities/math.py) as 'math_slope' to be
coherent with the new notation.

Change-Id: I02ccd2b87f0b72e7a28c416b593aae4d8ad97961
JIRA: STORPERF-57
JIRA: STORPERF-58
Signed-off-by: Tim Rault <tim.rault@cengn.ca>
8 years agoFix missing test 79/16879/1
Mark Beierl [Thu, 14 Jul 2016 04:11:23 +0000 (00:11 -0400)]
Fix missing test

All test files must end in _test.py or they don't get
executed

Change-Id: If3a46a66570f7d07d0e93de72438a8a46b3e0a22
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
8 years agoAdd Least Squares slope function 13/16213/3
Tim Rault [Wed, 29 Jun 2016 14:54:55 +0000 (10:54 -0400)]
Add Least Squares slope function

Added a math module in utilities that contains a slope function
able to compute the slope of the best Least Squares curve fit
given a series of [x,y] values.

Implemented a test harness for this math module in the tests/utilities
section.

Change-Id: If4d63af092d0904b2269c5ee0991e18ab84533c0
JIRA: STORPERF-54
JIRA: STORPERF-55
JIRA: STORPERF-51
Signed-off-by: Tim Rault <tim.rault@cengn.ca>
8 years agoForce a compatible version of Coverage 73/15773/1
Mark Beierl [Thu, 16 Jun 2016 19:51:41 +0000 (15:51 -0400)]
Force a compatible version of Coverage

LiClipse / PyDev only works with 3.4

Change-Id: I351df5f8120889af07b10ebb7a740d1e318cb1c2
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
8 years agoAutocreate the key pair 33/14733/2
Mark Beierl [Thu, 26 May 2016 15:14:05 +0000 (11:14 -0400)]
Autocreate the key pair

Change-Id: Ib4b8cb92b4cbc7dd778363335b6d0e22ae1993fb
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
8 years agoStandardize the ReST API 37/14437/1
Mark Beierl [Fri, 20 May 2016 13:30:46 +0000 (09:30 -0400)]
Standardize the ReST API

Change-Id: I004824711abf17f06781302cf48a0b0e3d095ecb
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoFixed logic indendation bug 97/14097/1
Mark Beierl [Fri, 13 May 2016 14:49:28 +0000 (08:49 -0600)]
Fixed logic indendation bug

Change-Id: I6cb17c699ae3b88a591278da23d60dfc0da91e40
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoFixed spacing for swagger docs 65/14065/1
Mark Beierl [Thu, 12 May 2016 21:57:32 +0000 (15:57 -0600)]
Fixed spacing for swagger docs

Change-Id: I994153d03981f9b706834ab2ab09196e235cc7ce
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoResults DB Reporting 53/14053/2
Mark Beierl [Thu, 12 May 2016 16:05:25 +0000 (10:05 -0600)]
Results DB Reporting

Add the ability to push results to the test result db.  If
the environment variable TEST_DB_URL is defined, results
will be pushed there.  If not, no push is attempted.

Change-Id: Ib833530d7379c5f37f0d2904a83d31a4ee559ae6
JIRA: STORPERF-13
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoMax test duration 65/13965/3
Mark Beierl [Wed, 11 May 2016 21:38:26 +0000 (15:38 -0600)]
Max test duration

Add a maximum time limit for any one given test so
that silly mistakes like block size 512, queue depth 1
does not end up running for 2 days

JIRA: STORPERF-53

Change-Id: I1f3e5046e1dd346fc5b28a7b935df2ee9aa6ba3d
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoJob Run Metadata 43/13943/2
Mark Beierl [Wed, 11 May 2016 16:54:48 +0000 (10:54 -0600)]
Job Run Metadata

Add the ability to store arbitrary metadata about the job for
later reporting

Change-Id: If7f1d0a69111567d69db865baec6c20f9a1f494f
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoFix double loop bug 79/13879/1
Mark Beierl [Tue, 10 May 2016 23:15:54 +0000 (17:15 -0600)]
Fix double loop bug

Change-Id: I47f20dec309d7ab8402f40f6cf09828aa134e902
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoAuth timeout fix and VM image selection 63/13863/1
Mark Beierl [Tue, 10 May 2016 16:48:53 +0000 (10:48 -0600)]
Auth timeout fix and VM image selection

Fix the OpenStack client authentication timeout bug where we never
reauthenticate after authentication expires
Add the ability to specify what OS image from the Glance catalog
should be used for the Agent VM

Change-Id: I2ff8f5598b8d58ee2bf654b3572e9841b24cc9de
JIRA: STORPERF-42
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoAdd plots 39/13739/2
Mark Beierl [Sun, 8 May 2016 14:09:47 +0000 (10:09 -0400)]
Add plots

Adds matplotlib
Adds support for generating graphs showing performance results

Change-Id: Iba344baab89e35c0d5891a9d6b4c22c37666511e
JIRA: STORPERF-44
JIRA: STORPERF-50
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years ago Add Stats report and Swagger UI 27/13727/3
Mark Beierl [Thu, 5 May 2016 02:53:07 +0000 (22:53 -0400)]
Add Stats report and Swagger UI

    Add Swagger web ui at /swagger
    Add ability to fetch read/write latency status via ReST ui
    Can now delete where stack was removed from OpenStack but not from the
    storperf DB
    Change to use Floating IPs instead of private IP
    Fix delete bug where there was no dependency on resources in
    the resource group.

    JIRA: STORPERF-19
    JIRA: STORPERF-20

Change-Id: I1d9627d81f3c309b178a9b68cc306a4101c1a231
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years ago Add Stats report and Swagger UI 71/13671/4
Mark Beierl [Thu, 5 May 2016 02:53:07 +0000 (22:53 -0400)]
Add Stats report and Swagger UI

    Add Swagger web ui at /swagger
    Add ability to fetch read/write latency status via ReST ui
    Can now delete where stack was removed from OpenStack but not from the
    storperf DB
    Change to use Floating IPs instead of private IP
    Fix delete bug where there was no dependency on resources in
    the resource group.

    JIRA: STORPERF-19
    JIRA: STORPERF-20

Change-Id: I0a4b3386789c38d6745906ba896b8ff851dc122f
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoFix flake8 violations 87/12687/1
jose.lausuch [Tue, 26 Apr 2016 19:45:45 +0000 (21:45 +0200)]
Fix flake8 violations

Change-Id: Idd0f4bec503578c2f2031cbcdee97ea27d22ac14
Signed-off-by: jose.lausuch <jose.lausuch@ericsson.com>
9 years agoCancel Job API 83/12683/2
Mark Beierl [Tue, 26 Apr 2016 17:18:23 +0000 (13:18 -0400)]
Cancel Job API

Add the ability to terminate a running job via the API

JIRA: STORPERF-20

Change-Id: I73a701cff9712207f5e14cfcc6b8fb7e0ab59aed
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoJob run lifecycle rework 37/12637/3
Mark Beierl [Mon, 25 Apr 2016 13:55:03 +0000 (09:55 -0400)]
Job run lifecycle rework

Change the way slave jobs are managed so that they are in step
with each other, and we can track the overall thread that is
running them.  This lays groundwork for STORPERF-20 and
STORPERF-44

JIRA: STORPERF-33 STORPERF-43

Change-Id: Iaff48a2823ba85d6512e9782fd9091a72639835c
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoTests in verify 63/12363/5
Mark Beierl [Fri, 15 Apr 2016 17:10:47 +0000 (13:10 -0400)]
Tests in verify

Run the tests in the verify command instead
of just merge so that way patches can be
rejected if tests fail

Change-Id: I8907618a324296460a71a51d087926f9089e0d21
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoTest Run Default Values 91/12091/1
Mark Beierl [Thu, 7 Apr 2016 20:54:23 +0000 (16:54 -0400)]
Test Run Default Values

Change test run to default to 64k blocks with 100% of attached storage
being tested.

JIRA: STORPERF-33

Change-Id: Id803d0493fb2c35691911ed79d226443a01503f3
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoGraphite module load fix - reopen 01/12001/1
Mark Beierl [Thu, 7 Apr 2016 14:17:53 +0000 (14:17 +0000)]
Graphite module load fix - reopen

Create symlinks for whisper so that Graphite
can find the modules at runtime. Needs to be
done after Graphite is installed

Change-Id: I383557138fa083d4ffeea307868465f72efaeb69
JIRA: STORPERF-41
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoIncorrect name 65/11965/1
Mark Beierl [Wed, 6 Apr 2016 20:15:22 +0000 (16:15 -0400)]
Incorrect name

StorPerf agent now uses storperf in ssh
HOT uses default name StorPerf Ubuntu 14.04

Change-Id: Ie87443955c73d1e81fb3a75b90e4a65a9ac6560b
JIRA: STORPERF-39
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoFixup initilization 83/11083/1
Mark Beierl [Wed, 9 Mar 2016 16:28:12 +0000 (11:28 -0500)]
Fixup initilization

Fetch the target from the correct field.
Initialize the workloads to a known value

Change-Id: Iadea5b580671fc2e4a1394248b800a446a37824a
JIRA: STORPERF-36 STORPERF-37
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoFIO needs sudo 69/11069/1
Mark Beierl [Wed, 9 Mar 2016 16:19:32 +0000 (11:19 -0500)]
FIO needs sudo

Use sudo for fio when profiling a device.

Change-Id: Ibbcbb15a2c33e1e58b7e340f9c3f6de5958f5636
JIRA: STORPERF-40
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoDeadlock in job_db 65/11065/1
Mark Beierl [Wed, 9 Mar 2016 15:27:35 +0000 (10:27 -0500)]
Deadlock in job_db

Change order of operation so that we do not attempt
to lock the same mutex twice in a row.

Change-Id: I1f874d720e2b66d9b272be0202482b22d24ef2b2
JIRA: STORPERF-42
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoMerge "Cinder fix"
Mark Beierl [Wed, 9 Mar 2016 14:55:49 +0000 (14:55 +0000)]
Merge "Cinder fix"

9 years agoCinder fix 31/11031/1
Mark Beierl [Tue, 8 Mar 2016 20:15:02 +0000 (15:15 -0500)]
Cinder fix

Fix so that Cinder will use Tenant ID instead of name

Change-Id: I2d30db61dffc4f381d64e8d99449a2c55c5928a9
JIRA: STORPERF-38
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoGraphite module load fix 29/11029/1
Mark Beierl [Tue, 8 Mar 2016 19:21:39 +0000 (14:21 -0500)]
Graphite module load fix

Create symlinks for whisper so that Graphite
can find the modules at runtime.

Change-Id: Ie0defb180a29ba430978be091d6683a6cf7d5891
JIRA: STORPERF-41
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoDB Access Mutex 47/10647/1
Mark Beierl [Wed, 24 Feb 2016 22:04:14 +0000 (23:04 +0100)]
DB Access Mutex

Addition of a mutex to prevent mutiple, simultaneous DB accesses as SQLite is single threaded

Change-Id: Iad2cd94015f7fb604dc6b60636db939099f52757
JIRA: STORPERF-32
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoDeveloper build process fix 99/9199/1
Mark Beierl [Wed, 3 Feb 2016 22:18:54 +0000 (17:18 -0500)]
Developer build process fix

Fix the installation os setuptools so that it
does not conflict with mock.

Change-Id: I7c497d36c80a795fc99b329da2d171a265e4eacc
JIRA: STORPERF-35
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoFixed typo 01/8701/1
Mark Beierl [Fri, 29 Jan 2016 22:19:25 +0000 (17:19 -0500)]
Fixed typo

SSHD failed to start due to typo in supervisord.conf

Change-Id: I23fd1452b83dd28d1c2c24760a5d428e01f90799
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoRemote slave agent workload 65/7365/4
Mark Beierl [Wed, 20 Jan 2016 01:58:35 +0000 (20:58 -0500)]
Remote slave agent workload

Add storperf master object to manage stack
lifecycle.
Add configuration db.
Creation of CLI vs. main so that ReST API
and CLI API can be kept clear.
Fixed License in files.
Changed DB objects to be thread safe.
Added ssh server to container if desired
for CLI.

Change-Id: Idfe84bfb7920e6ce19d27462593c21ea86e7b406
JIRA: STORPERF-29
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoSwitching merge job to virtualenv 15/5815/1
Mark Beierl [Thu, 7 Jan 2016 17:17:03 +0000 (12:17 -0500)]
Switching merge job to virtualenv

Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoWorkload reporting 77/4377/1
mbeierl [Fri, 11 Dec 2015 20:31:17 +0000 (15:31 -0500)]
Workload reporting

Use a local db to track start/end times of runs so we
can go back to the carbon db and summarize values
at reporting time based off the raw data.

Change-Id: Ie62afd339fd1c15d82bc56c93c7cba5bd4f90fe2
JIRA: STORPERF-29
Signed-off-by: mbeierl <mark.beierl@emc.com>
9 years agoAdditional FIO and development build support 31/4231/1
Mark Beierl [Thu, 10 Dec 2015 21:14:43 +0000 (16:14 -0500)]
Additional FIO and development build support

Add async IO development package so that FIO can
take advantage of that in order to drive queue
depth up.
Add supervisord to container so graphite and
carbon can start automatically
Add a developer build script so we can create
a container based on current state of local
repo

Change-Id: Ibd57f484b2fa41aa48a1c4fb06217e7d6e1c0b3a
JIRA: STORPERF-15
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoAdding Jose Lausuch as commiter 07/4207/1
Mark Beierl [Thu, 10 Dec 2015 16:49:51 +0000 (11:49 -0500)]
Adding Jose Lausuch as commiter

Change-Id: I22c24e2bfd8f08ce4ff7f18794280582f564b8f0
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoCreate dockerfile to build the storperf image 91/4191/6
jose.lausuch [Thu, 10 Dec 2015 13:50:56 +0000 (14:50 +0100)]
Create dockerfile to build the storperf image

JIRA: STORPERF-15

Change-Id: I1e35694e8b091caa3718c341f9f95217e5b0b609
Signed-off-by: jose.lausuch <jose.lausuch@ericsson.com>
9 years agoThis should not be under source code control 89/4189/2
Mark Beierl [Thu, 10 Dec 2015 13:37:54 +0000 (08:37 -0500)]
This should not be under source code control

Change-Id: Ia4f84fef338401604a704cd5b690b7bddd968886
Signed-off-by: Mark Beierl <mark.beierl@emc.com>
9 years agoAdding workload modules 47/3547/2
mbeierl [Mon, 23 Nov 2015 16:23:47 +0000 (08:23 -0800)]
Adding workload modules

Adding the ablity to define workloads in modules which can be
referenced from the API.  Breaking out the test execution into
its own class so it will be easier to support ReST or other
interfaces.

Added flake8 and code coverage reports where possible to
merge and verify jobs

Change-Id: Ieb51e4e7e1e989288a6f81f4757709669914a196
JIRA: STORPERF-21
Signed-off-by: mbeierl <mark.beierl@emc.com>
9 years agoAdding Heat template for staging StorPerf master node 89/3089/1
mbeierl [Fri, 6 Nov 2015 02:22:54 +0000 (21:22 -0500)]
Adding Heat template for staging StorPerf master node

Change-Id: I1e55290013f31272952f23869a02d68ca08cd4a7
JIRA: STORPERF-12
Signed-off-by: mbeierl <mark.beierl@emc.com>
9 years agoIntegration with Releng 05/2605/1
Mark Beierl [Fri, 16 Oct 2015 20:42:10 +0000 (16:42 -0400)]
Integration with Releng

Added verify, merge and daily jobs that will be
called automatically by Gerrit/LF Jenkins on
workflow change

Change-Id: I3ae85593b6aeac4a1a7abb51354b9cc6fa1fd813
JIRA: RELENG-41
Signed-off-by: Mark Beierl <mark@jemms.net>
9 years agoLogging and timestamp 63/2563/4
mbeierl [Thu, 15 Oct 2015 14:48:58 +0000 (10:48 -0400)]
Logging and timestamp

Added logging and fixed issue if fio does not produce the current timestamp
in its output

JIRA: STORPERF-4

Change-Id: Ifd0dbc4e17d984907e63089ebfae1d0e9e749dcc
Signed-off-by: mbeierl <mark.beierl@emc.com>
9 years agoMain and fio interaction 63/2363/5
mbeierl [Thu, 8 Oct 2015 02:20:05 +0000 (19:20 -0700)]
Main and fio interaction

JIRA: STORPERF-4

Change-Id: I60efb773848459f994662db8f050174ab9c86bd0
Signed-off-by: mbeierl <mark.beierl@emc.com>
9 years agoCreation of converters 61/2361/4
mbeierl [Thu, 8 Oct 2015 02:19:22 +0000 (19:19 -0700)]
Creation of converters

JIRA: STORPERF-6

Change-Id: Id81a129807a984ad4af669e3f0f84c1e7ced5552
Signed-off-by: mbeierl <mark.beierl@emc.com>
9 years agoCreation of emitters 59/2359/4
mbeierl [Thu, 8 Oct 2015 02:19:11 +0000 (19:19 -0700)]
Creation of emitters

JIRA: STORPERF-7

Change-Id: I0a34af62ef850c9fc3e117b5c53283fc8d17561f
Signed-off-by: mbeierl <mark.beierl@emc.com>
9 years agoCreated converters 57/2357/4
mbeierl [Thu, 8 Oct 2015 02:18:19 +0000 (19:18 -0700)]
Created converters

JIRA: STORPERF-6

Change-Id: Idba1718a8ad8243872ca100ed10b1c457221d147
Signed-off-by: mbeierl <mark.beierl@emc.com>
9 years agoAdded PyDev projects 55/2355/4
mbeierl [Thu, 8 Oct 2015 02:14:29 +0000 (19:14 -0700)]
Added PyDev projects

JIRA: STORPERF-1

Change-Id: I1bf9ff0d7f512d3441ac23795fbe80b4884795dd
Signed-off-by: mbeierl <mark.beierl@emc.com>
9 years agoAdded Jenkins scripts 53/2353/3
mbeierl [Thu, 8 Oct 2015 02:12:20 +0000 (19:12 -0700)]
Added Jenkins scripts
JIRA: STORPERF-12
JIRA: STORPERF-1

Change-Id: I222381a690c89324058c8b6510b33a94c0ce0052
Signed-off-by: mbeierl <mark.beierl@emc.com>
9 years agoAdding INFO and LICENSE for review 97/1597/1
Aric Gardner [Tue, 15 Sep 2015 16:57:39 +0000 (12:57 -0400)]
Adding INFO and LICENSE for review

Change-Id: Ic565f117347ed80a5aee703254224c6f4b57b80e
Signed-off-by: Aric Gardner <agardner@linuxfoundation.org>
9 years agoInitial empty repository
Aric Gardner [Tue, 15 Sep 2015 16:57:31 +0000 (16:57 +0000)]
Initial empty repository