Initiate packetization of Testing reporting 29/36929/11
authorMorgan Richomme <morgan.richomme@orange.com>
Tue, 4 Jul 2017 15:23:33 +0000 (17:23 +0200)
committerMorgan Richomme <morgan.richomme@orange.com>
Wed, 9 Aug 2017 14:45:43 +0000 (16:45 +0200)
Testing reporting provides
- static reporting pages [1] (jinja2)
- landing pages including test case catalogue [2] (angular)

It consumes the Test API to build web pages providing status for
the testing projects (so far functest, yardstick, storperf and
qtip).

Dockerization has been initiated [3]. The goal is to replace the
static page hosted on testresults.opnfv.org by a docker regenerated
and redeployed when reporting code is changed

But the docker and more generally the testing reporting directory
must be refactored to
- manage dependencies properly
- leverage tox (py27, docs, pylint, pep8)

This patch
- setups tox (py27, pep8, pylint, docs)
- integrate the requirement management
- fix pep8 errors
- introduce 1 dummy unit test

This patch does not
- fix pylint errors
- create any doc

[1]: http://testresults.opnfv.org/reporting2/display/index.html
[2]: http://testresults.opnfv.org/reporting2/reporting/index.html
[3]: https://gerrit.opnfv.org/gerrit/#/c/36735/

Change-Id: I4613de7ca7036d6c6bbb8f58ade492b1d673599b
Signed-off-by: Morgan Richomme <morgan.richomme@orange.com>
96 files changed:
utils/test/reporting/api/__init__.py [moved from utils/test/reporting/api/api/__init__.py with 100% similarity]
utils/test/reporting/api/conf.py [moved from utils/test/reporting/api/api/conf.py with 100% similarity]
utils/test/reporting/api/handlers/__init__.py [moved from utils/test/reporting/api/api/handlers/__init__.py with 100% similarity]
utils/test/reporting/api/handlers/landing.py [moved from utils/test/reporting/api/api/handlers/landing.py with 100% similarity]
utils/test/reporting/api/handlers/projects.py [moved from utils/test/reporting/api/api/handlers/projects.py with 100% similarity]
utils/test/reporting/api/handlers/testcases.py [moved from utils/test/reporting/api/api/handlers/testcases.py with 100% similarity]
utils/test/reporting/api/requirements.txt [deleted file]
utils/test/reporting/api/server.py [moved from utils/test/reporting/api/api/server.py with 100% similarity]
utils/test/reporting/api/setup.cfg [deleted file]
utils/test/reporting/api/setup.py [deleted file]
utils/test/reporting/api/urls.py [moved from utils/test/reporting/api/api/urls.py with 100% similarity]
utils/test/reporting/docker/Dockerfile
utils/test/reporting/docker/nginx.conf
utils/test/reporting/docker/reporting.sh
utils/test/reporting/docker/supervisor.conf
utils/test/reporting/docker/web_server.sh [new file with mode: 0755]
utils/test/reporting/docs/_build/.buildinfo [new file with mode: 0644]
utils/test/reporting/docs/_build/.doctrees/environment.pickle [new file with mode: 0644]
utils/test/reporting/docs/_build/.doctrees/index.doctree [new file with mode: 0644]
utils/test/reporting/docs/conf.py [new file with mode: 0644]
utils/test/reporting/docs/index.rst [new file with mode: 0644]
utils/test/reporting/reporting/__init__.py [moved from utils/test/reporting/functest/__init__.py with 100% similarity]
utils/test/reporting/reporting/functest/__init__.py [moved from utils/test/reporting/qtip/__init__.py with 100% similarity]
utils/test/reporting/reporting/functest/img/gauge_0.png [moved from utils/test/reporting/functest/img/gauge_0.png with 100% similarity]
utils/test/reporting/reporting/functest/img/gauge_100.png [moved from utils/test/reporting/functest/img/gauge_100.png with 100% similarity]
utils/test/reporting/reporting/functest/img/gauge_16.7.png [moved from utils/test/reporting/functest/img/gauge_16.7.png with 100% similarity]
utils/test/reporting/reporting/functest/img/gauge_25.png [moved from utils/test/reporting/functest/img/gauge_25.png with 100% similarity]
utils/test/reporting/reporting/functest/img/gauge_33.3.png [moved from utils/test/reporting/functest/img/gauge_33.3.png with 100% similarity]
utils/test/reporting/reporting/functest/img/gauge_41.7.png [moved from utils/test/reporting/functest/img/gauge_41.7.png with 100% similarity]
utils/test/reporting/reporting/functest/img/gauge_50.png [moved from utils/test/reporting/functest/img/gauge_50.png with 100% similarity]
utils/test/reporting/reporting/functest/img/gauge_58.3.png [moved from utils/test/reporting/functest/img/gauge_58.3.png with 100% similarity]
utils/test/reporting/reporting/functest/img/gauge_66.7.png [moved from utils/test/reporting/functest/img/gauge_66.7.png with 100% similarity]
utils/test/reporting/reporting/functest/img/gauge_75.png [moved from utils/test/reporting/functest/img/gauge_75.png with 100% similarity]
utils/test/reporting/reporting/functest/img/gauge_8.3.png [moved from utils/test/reporting/functest/img/gauge_8.3.png with 100% similarity]
utils/test/reporting/reporting/functest/img/gauge_83.3.png [moved from utils/test/reporting/functest/img/gauge_83.3.png with 100% similarity]
utils/test/reporting/reporting/functest/img/gauge_91.7.png [moved from utils/test/reporting/functest/img/gauge_91.7.png with 100% similarity]
utils/test/reporting/reporting/functest/img/icon-nok.png [moved from utils/test/reporting/functest/img/icon-nok.png with 100% similarity]
utils/test/reporting/reporting/functest/img/icon-ok.png [moved from utils/test/reporting/functest/img/icon-ok.png with 100% similarity]
utils/test/reporting/reporting/functest/img/weather-clear.png [moved from utils/test/reporting/functest/img/weather-clear.png with 100% similarity]
utils/test/reporting/reporting/functest/img/weather-few-clouds.png [moved from utils/test/reporting/functest/img/weather-few-clouds.png with 100% similarity]
utils/test/reporting/reporting/functest/img/weather-overcast.png [moved from utils/test/reporting/functest/img/weather-overcast.png with 100% similarity]
utils/test/reporting/reporting/functest/img/weather-storm.png [moved from utils/test/reporting/functest/img/weather-storm.png with 100% similarity]
utils/test/reporting/reporting/functest/index.html [moved from utils/test/reporting/functest/index.html with 100% similarity]
utils/test/reporting/reporting/functest/reporting-status.py [moved from utils/test/reporting/functest/reporting-status.py with 98% similarity]
utils/test/reporting/reporting/functest/reporting-tempest.py [moved from utils/test/reporting/functest/reporting-tempest.py with 98% similarity]
utils/test/reporting/reporting/functest/reporting-vims.py [moved from utils/test/reporting/functest/reporting-vims.py with 97% similarity]
utils/test/reporting/reporting/functest/scenarioResult.py [moved from utils/test/reporting/functest/scenarioResult.py with 100% similarity]
utils/test/reporting/reporting/functest/template/index-status-tmpl.html [moved from utils/test/reporting/functest/template/index-status-tmpl.html with 100% similarity]
utils/test/reporting/reporting/functest/template/index-tempest-tmpl.html [moved from utils/test/reporting/functest/template/index-tempest-tmpl.html with 100% similarity]
utils/test/reporting/reporting/functest/template/index-vims-tmpl.html [moved from utils/test/reporting/functest/template/index-vims-tmpl.html with 100% similarity]
utils/test/reporting/reporting/functest/testCase.py [moved from utils/test/reporting/functest/testCase.py with 100% similarity]
utils/test/reporting/reporting/qtip/__init__.py [moved from utils/test/reporting/tests/__init__.py with 100% similarity]
utils/test/reporting/reporting/qtip/index.html [moved from utils/test/reporting/qtip/index.html with 100% similarity]
utils/test/reporting/reporting/qtip/reporting-status.py [moved from utils/test/reporting/qtip/reporting-status.py with 93% similarity]
utils/test/reporting/reporting/qtip/template/index-status-tmpl.html [moved from utils/test/reporting/qtip/template/index-status-tmpl.html with 100% similarity]
utils/test/reporting/reporting/reporting.yaml [moved from utils/test/reporting/reporting.yaml with 100% similarity]
utils/test/reporting/reporting/storperf/__init__.py [moved from utils/test/reporting/tests/unit/__init__.py with 100% similarity]
utils/test/reporting/reporting/storperf/reporting-status.py [moved from utils/test/reporting/storperf/reporting-status.py with 98% similarity]
utils/test/reporting/reporting/storperf/template/index-status-tmpl.html [moved from utils/test/reporting/storperf/template/index-status-tmpl.html with 100% similarity]
utils/test/reporting/reporting/tests/__init__.py [moved from utils/test/reporting/tests/unit/utils/__init__.py with 100% similarity]
utils/test/reporting/reporting/tests/unit/__init__.py [moved from utils/test/reporting/utils/__init__.py with 100% similarity]
utils/test/reporting/reporting/tests/unit/utils/__init__.py [new file with mode: 0644]
utils/test/reporting/reporting/tests/unit/utils/test_utils.py [moved from utils/test/reporting/tests/unit/utils/test_utils.py with 78% similarity]
utils/test/reporting/reporting/utils/__init__.py [new file with mode: 0644]
utils/test/reporting/reporting/utils/reporting_utils.py [moved from utils/test/reporting/utils/reporting_utils.py with 92% similarity]
utils/test/reporting/reporting/utils/scenarioResult.py [moved from utils/test/reporting/utils/scenarioResult.py with 100% similarity]
utils/test/reporting/reporting/yardstick/__init__.py [new file with mode: 0644]
utils/test/reporting/reporting/yardstick/img/gauge_0.png [moved from utils/test/reporting/yardstick/img/gauge_0.png with 100% similarity]
utils/test/reporting/reporting/yardstick/img/gauge_100.png [moved from utils/test/reporting/yardstick/img/gauge_100.png with 100% similarity]
utils/test/reporting/reporting/yardstick/img/gauge_16.7.png [moved from utils/test/reporting/yardstick/img/gauge_16.7.png with 100% similarity]
utils/test/reporting/reporting/yardstick/img/gauge_25.png [moved from utils/test/reporting/yardstick/img/gauge_25.png with 100% similarity]
utils/test/reporting/reporting/yardstick/img/gauge_33.3.png [moved from utils/test/reporting/yardstick/img/gauge_33.3.png with 100% similarity]
utils/test/reporting/reporting/yardstick/img/gauge_41.7.png [moved from utils/test/reporting/yardstick/img/gauge_41.7.png with 100% similarity]
utils/test/reporting/reporting/yardstick/img/gauge_50.png [moved from utils/test/reporting/yardstick/img/gauge_50.png with 100% similarity]
utils/test/reporting/reporting/yardstick/img/gauge_58.3.png [moved from utils/test/reporting/yardstick/img/gauge_58.3.png with 100% similarity]
utils/test/reporting/reporting/yardstick/img/gauge_66.7.png [moved from utils/test/reporting/yardstick/img/gauge_66.7.png with 100% similarity]
utils/test/reporting/reporting/yardstick/img/gauge_75.png [moved from utils/test/reporting/yardstick/img/gauge_75.png with 100% similarity]
utils/test/reporting/reporting/yardstick/img/gauge_8.3.png [moved from utils/test/reporting/yardstick/img/gauge_8.3.png with 100% similarity]
utils/test/reporting/reporting/yardstick/img/gauge_83.3.png [moved from utils/test/reporting/yardstick/img/gauge_83.3.png with 100% similarity]
utils/test/reporting/reporting/yardstick/img/gauge_91.7.png [moved from utils/test/reporting/yardstick/img/gauge_91.7.png with 100% similarity]
utils/test/reporting/reporting/yardstick/img/icon-nok.png [moved from utils/test/reporting/yardstick/img/icon-nok.png with 100% similarity]
utils/test/reporting/reporting/yardstick/img/icon-ok.png [moved from utils/test/reporting/yardstick/img/icon-ok.png with 100% similarity]
utils/test/reporting/reporting/yardstick/img/weather-clear.png [moved from utils/test/reporting/yardstick/img/weather-clear.png with 100% similarity]
utils/test/reporting/reporting/yardstick/img/weather-few-clouds.png [moved from utils/test/reporting/yardstick/img/weather-few-clouds.png with 100% similarity]
utils/test/reporting/reporting/yardstick/img/weather-overcast.png [moved from utils/test/reporting/yardstick/img/weather-overcast.png with 100% similarity]
utils/test/reporting/reporting/yardstick/img/weather-storm.png [moved from utils/test/reporting/yardstick/img/weather-storm.png with 100% similarity]
utils/test/reporting/reporting/yardstick/index.html [moved from utils/test/reporting/yardstick/index.html with 100% similarity]
utils/test/reporting/reporting/yardstick/reporting-status.py [moved from utils/test/reporting/yardstick/reporting-status.py with 98% similarity]
utils/test/reporting/reporting/yardstick/scenarios.py [moved from utils/test/reporting/yardstick/scenarios.py with 100% similarity]
utils/test/reporting/reporting/yardstick/template/index-status-tmpl.html [moved from utils/test/reporting/yardstick/template/index-status-tmpl.html with 100% similarity]
utils/test/reporting/requirements.txt [new file with mode: 0644]
utils/test/reporting/run_test.sh
utils/test/reporting/setup.cfg [new file with mode: 0644]
utils/test/reporting/setup.py
utils/test/reporting/test-requirements.txt [new file with mode: 0644]
utils/test/reporting/tox.ini [new file with mode: 0644]

diff --git a/utils/test/reporting/api/requirements.txt b/utils/test/reporting/api/requirements.txt
deleted file mode 100644 (file)
index 12ad688..0000000
+++ /dev/null
@@ -1,3 +0,0 @@
-tornado==4.4.2
-requests==2.1.0
-
diff --git a/utils/test/reporting/api/setup.cfg b/utils/test/reporting/api/setup.cfg
deleted file mode 100644 (file)
index 53d1092..0000000
+++ /dev/null
@@ -1,32 +0,0 @@
-[metadata]
-name = reporting
-
-author = JackChan
-author-email = chenjiankun1@huawei.com
-
-classifier =
-    Environment :: opnfv
-    Intended Audience :: Information Technology
-    Intended Audience :: System Administrators
-    License :: OSI Approved :: Apache Software License
-    Operating System :: POSIX :: Linux
-    Programming Language :: Python
-    Programming Language :: Python :: 2
-    Programming Language :: Python :: 2.7
-
-[global]
-setup-hooks =
-    pbr.hooks.setup_hook
-
-[files]
-packages =
-    api
-
-[entry_points]
-console_scripts =
-    api = api.server:main
-
-[egg_info]
-tag_build =
-tag_date = 0
-tag_svn_revision = 0
diff --git a/utils/test/reporting/api/setup.py b/utils/test/reporting/api/setup.py
deleted file mode 100644 (file)
index d974816..0000000
+++ /dev/null
@@ -1,9 +0,0 @@
-import setuptools
-
-
-__author__ = 'JackChan'
-
-
-setuptools.setup(
-    setup_requires=['pbr>=1.8'],
-    pbr=True)
index ad278ce..f5168d1 100644 (file)
 FROM nginx:stable
 
 MAINTAINER Morgan Richomme <morgan.richomme@orange.com>
-LABEL version="danube.1.0" description="OPNFV Test Reporting Docker container"
+LABEL version="1.0" description="OPNFV Test Reporting Docker container"
 
 ARG BRANCH=master
 
 ENV HOME /home/opnfv
-ENV working_dir /home/opnfv/utils/test/reporting
-ENV TERM xterm
-ENV COLORTERM gnome-terminal
-ENV CONFIG_REPORTING_YAML /home/opnfv/utils/test/reporting/reporting.yaml
+ENV working_dir ${HOME}/releng/utils/test/reporting
+ENV CONFIG_REPORTING_YAML ${working_dir}/reporting.yaml
 
+WORKDIR ${HOME}
 # Packaged dependencies
 RUN apt-get update && apt-get install -y \
 ssh \
 python-pip \
 git-core \
-wkhtmltopdf \
 nodejs \
 npm \
 supervisor \
@@ -39,15 +37,13 @@ supervisor \
 
 RUN pip install --upgrade pip
 
-RUN git clone --depth 1 https://gerrit.opnfv.org/gerrit/releng /home/opnfv
-RUN pip install -r ${working_dir}/docker/requirements.pip
-
-WORKDIR ${working_dir}/api
-RUN pip install -r requirements.txt
-RUN python setup.py install
+RUN git clone --depth 1 https://gerrit.opnfv.org/gerrit/releng ${HOME}/releng
+RUN pip install -r ${working_dir}/requirements.txt
 
 WORKDIR ${working_dir}
+RUN python setup.py install
 RUN docker/reporting.sh
+RUN docker/web_server.sh
 
 expose 8000
 
index 9e26972..ced8179 100644 (file)
@@ -15,10 +15,10 @@ server {
     }
 
     location /reporting/ {
-        alias /home/opnfv/utils/test/reporting/pages/dist/;
+        alias /home/opnfv/releng/utils/test/reporting/pages/dist/;
     }
 
     location /display/ {
-        alias /home/opnfv/utils/test/reporting/display/;
+        alias /home/opnfv/releng/utils/test/reporting/display/;
     }
 }
index 7fe97a8..076dc47 100755 (executable)
@@ -1,10 +1,10 @@
 #!/bin/bash
 
-export PYTHONPATH="${PYTHONPATH}:."
-export CONFIG_REPORTING_YAML=./reporting.yaml
+export PYTHONPATH="${PYTHONPATH}:./reporting"
+export CONFIG_REPORTING_YAML=./reporting/reporting.yaml
 
 declare -a versions=(danube master)
-declare -a projects=(functest storperf yardstick)
+declare -a projects=(functest storperf yardstick qtip)
 
 project=$1
 reporting_type=$2
@@ -29,8 +29,9 @@ cp -Rf js display
 #  projet   |        option
 #   $1      |          $2
 # functest  | status, vims, tempest
-# yardstick |
-# storperf  |
+# yardstick | status
+# storperf  | status
+# qtip      | status
 
 function report_project()
 {
@@ -40,7 +41,7 @@ function report_project()
   echo "********************************"
   echo " $project reporting "
   echo "********************************"
-  python ./$dir/reporting-$type.py
+  python ./reporting/$dir/reporting-$type.py
   if [ $? ]; then
     echo "$project reporting $type...OK"
   else
@@ -50,53 +51,28 @@ function report_project()
 
 if [ -z "$1" ]; then
   echo "********************************"
-  echo " Functest reporting "
+  echo " * Static status reporting     *"
   echo "********************************"
-  echo "reporting vIMS..."
-  python ./functest/reporting-vims.py
-  echo "reporting vIMS...OK"
-  sleep 10
-  echo "reporting Tempest..."
-  python ./functest/reporting-tempest.py
-  echo "reporting Tempest...OK"
-  sleep 10
-  echo "reporting status..."
-  python ./functest/reporting-status.py
-  echo "Functest reporting status...OK"
-
-  echo "********************************"
-  echo " Yardstick reporting "
-  echo "********************************"
-  python ./yardstick/reporting-status.py
-  echo "Yardstick reporting status...OK"
+  for i in "${projects[@]}"
+  do
+    report_project $i $i "status"
+    sleep 5
+  done
+  report_project "QTIP" "qtip" "status"
 
-  echo "********************************"
-  echo " Storperf reporting "
-  echo "********************************"
-  python ./storperf/reporting-status.py
-  echo "Storperf reporting status...OK"
 
-  report_project "QTIP" "qtip" "status"
+  echo "Functest reporting vIMS..."
+  report_project "functest" "functest" "vims"
+  echo "reporting vIMS...OK"
+  sleep 5
+  echo "Functest reporting Tempest..."
+  report_project "functest" "functest" "tempest"
+  echo "reporting Tempest...OK"
+  sleep 5
 
 else
   if [ -z "$2" ]; then
     reporting_type="status"
   fi
-  echo "********************************"
-  echo " $project/$reporting_type reporting "
-  echo "********************************"
-  python ./$project/reporting-$reporting_type.py
+  report_project $project $project $reporting_type
 fi
-cp -r display /usr/share/nginx/html
-
-
-# nginx config
-cp /home/opnfv/utils/test/reporting/docker/nginx.conf /etc/nginx/conf.d/
-echo "daemon off;" >> /etc/nginx/nginx.conf
-
-# supervisor config
-cp /home/opnfv/utils/test/reporting/docker/supervisor.conf /etc/supervisor/conf.d/
-
-ln -s /usr/bin/nodejs /usr/bin/node
-
-cd pages && /bin/bash angular.sh
index b323dd0..49310d4 100644 (file)
@@ -3,7 +3,7 @@ nodaemon = true
 
 [program:tornado]
 user = root
-directory = /home/opnfv/utils/test/reporting/api/api
+directory = /home/opnfv/releng/utils/test/reporting/api
 command = python server.py --port=800%(process_num)d
 process_name=%(program_name)s%(process_num)d
 numprocs=4
@@ -15,5 +15,5 @@ command = service nginx restart
 
 [program:configuration]
 user = root
-directory = /home/opnfv/utils/test/reporting/pages
+directory = /home/opnfv/releng/utils/test/reporting/pages
 command = bash config.sh
diff --git a/utils/test/reporting/docker/web_server.sh b/utils/test/reporting/docker/web_server.sh
new file mode 100755 (executable)
index 0000000..a34c11d
--- /dev/null
@@ -0,0 +1,16 @@
+#!/bin/bash
+cp -r display /usr/share/nginx/html
+
+
+# nginx config
+cp /home/opnfv/releng/utils/test/reporting/docker/nginx.conf /etc/nginx/conf.d/
+echo "daemon off;" >> /etc/nginx/nginx.conf
+
+# supervisor config
+cp /home/opnfv/releng/utils/test/reporting/docker/supervisor.conf /etc/supervisor/conf.d/
+
+ln -s /usr/bin/nodejs /usr/bin/node
+
+# Manage Angular front end
+cd pages && /bin/bash angular.sh
+
diff --git a/utils/test/reporting/docs/_build/.buildinfo b/utils/test/reporting/docs/_build/.buildinfo
new file mode 100644 (file)
index 0000000..6bd6fd6
--- /dev/null
@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: 235ce07a48cec983846ad34dfd375b07
+tags: 645f666f9bcd5a90fca523b33c5a78b7
diff --git a/utils/test/reporting/docs/_build/.doctrees/environment.pickle b/utils/test/reporting/docs/_build/.doctrees/environment.pickle
new file mode 100644 (file)
index 0000000..23f59c3
Binary files /dev/null and b/utils/test/reporting/docs/_build/.doctrees/environment.pickle differ
diff --git a/utils/test/reporting/docs/_build/.doctrees/index.doctree b/utils/test/reporting/docs/_build/.doctrees/index.doctree
new file mode 100644 (file)
index 0000000..51e2d5a
Binary files /dev/null and b/utils/test/reporting/docs/_build/.doctrees/index.doctree differ
diff --git a/utils/test/reporting/docs/conf.py b/utils/test/reporting/docs/conf.py
new file mode 100644 (file)
index 0000000..2e70d2b
--- /dev/null
@@ -0,0 +1,341 @@
+# -*- coding: utf-8 -*-
+#
+# OPNFV testing Reporting documentation build configuration file, created by
+# sphinx-quickstart on Mon July 4 10:03:43 2017.
+#
+# This file is execfile()d with the current directory set to its
+# containing dir.
+#
+# Note that not all possible configuration values are present in this
+# autogenerated file.
+#
+# All configuration values have a default; values that are commented out
+# serve to show the default.
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+# import os
+# import sys
+# sys.path.insert(0, os.path.abspath('.'))
+
+# -- General configuration ------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+#
+# needs_sphinx = '1.0'
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    'sphinx.ext.autodoc',
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+#
+# source_suffix = ['.rst', '.md']
+source_suffix = '.rst'
+
+# The encoding of source files.
+#
+# source_encoding = 'utf-8-sig'
+
+# The master toctree document.
+master_doc = 'index'
+
+# General information about the project.
+project = u'OPNFV Reporting'
+copyright = u'2017, #opnfv-testperf (chat.freenode.net)'
+author = u'#opnfv-testperf (chat.freenode.net)'
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+# The short X.Y version.
+version = u'master'
+# The full version, including alpha/beta/rc tags.
+release = u'master'
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = 'en'
+
+# There are two options for replacing |today|: either, you set today to some
+# non-false value, then it is used:
+#
+# today = ''
+#
+# Else, today_fmt is used as the format for a strftime call.
+#
+# today_fmt = '%B %d, %Y'
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This patterns also effect to html_static_path and html_extra_path
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+# The reST default role (used for this markup: `text`) to use for all
+# documents.
+#
+# default_role = None
+
+# If true, '()' will be appended to :func: etc. cross-reference text.
+#
+# add_function_parentheses = True
+
+# If true, the current module name will be prepended to all description
+# unit titles (such as .. function::).
+#
+# add_module_names = True
+
+# If true, sectionauthor and moduleauthor directives will be shown in the
+# output. They are ignored by default.
+#
+# show_authors = False
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = 'sphinx'
+
+# A list of ignored prefixes for module index sorting.
+# modindex_common_prefix = []
+
+# If true, keep warnings as "system message" paragraphs in the built documents.
+# keep_warnings = False
+
+# If true, `todo` and `todoList` produce output, else they produce nothing.
+todo_include_todos = False
+
+
+# -- Options for HTML output ----------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'alabaster'
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+#
+# html_theme_options = {}
+
+# Add any paths that contain custom themes here, relative to this directory.
+# html_theme_path = []
+
+# The name for this set of Sphinx documents.
+# "<project> v<release> documentation" by default.
+#
+# html_title = u'OPNFV Functest vmaster'
+
+# A shorter title for the navigation bar.  Default is the same as html_title.
+#
+# html_short_title = None
+
+# The name of an image file (relative to this directory) to place at the top
+# of the sidebar.
+#
+# html_logo = None
+
+# The name of an image file (relative to this directory) to use as a favicon of
+# the docs.  This file should be a Windows icon file (.ico) being 16x16 or
+# 32x32 pixels large.
+#
+# html_favicon = None
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+# Add any extra paths that contain custom files (such as robots.txt or
+# .htaccess) here, relative to this directory. These files are copied
+# directly to the root of the documentation.
+#
+# html_extra_path = []
+
+# If not None, a 'Last updated on:' timestamp is inserted at every page
+# bottom, using the given strftime format.
+# The empty string is equivalent to '%b %d, %Y'.
+#
+# html_last_updated_fmt = None
+
+# If true, SmartyPants will be used to convert quotes and dashes to
+# typographically correct entities.
+#
+# html_use_smartypants = True
+
+# Custom sidebar templates, maps document names to template names.
+#
+# html_sidebars = {}
+
+# Additional templates that should be rendered to pages, maps page names to
+# template names.
+#
+# html_additional_pages = {}
+
+# If false, no module index is generated.
+#
+# html_domain_indices = True
+
+# If false, no index is generated.
+#
+# html_use_index = True
+
+# If true, the index is split into individual pages for each letter.
+#
+# html_split_index = False
+
+# If true, links to the reST sources are added to the pages.
+#
+# html_show_sourcelink = True
+
+# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
+#
+# html_show_sphinx = True
+
+# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
+#
+# html_show_copyright = True
+
+# If true, an OpenSearch description file will be output, and all pages will
+# contain a <link> tag referring to it.  The value of this option must be the
+# base URL from which the finished HTML is served.
+#
+# html_use_opensearch = ''
+
+# This is the file name suffix for HTML files (e.g. ".xhtml").
+# html_file_suffix = None
+
+# Language to be used for generating the HTML full-text search index.
+# Sphinx supports the following languages:
+#   'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja'
+#   'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr', 'zh'
+#
+# html_search_language = 'en'
+
+# A dictionary with options for the search language support, empty by default.
+# 'ja' uses this config value.
+# 'zh' user can custom change `jieba` dictionary path.
+#
+# html_search_options = {'type': 'default'}
+
+# The name of a javascript file (relative to the configuration directory) that
+# implements a search results scorer. If empty, the default will be used.
+#
+# html_search_scorer = 'scorer.js'
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = 'OPNFVreportingdoc'
+
+# -- Options for LaTeX output ---------------------------------------------
+
+latex_elements = {
+     # The paper size ('letterpaper' or 'a4paper').
+     #
+     # 'papersize': 'letterpaper',
+
+     # The font size ('10pt', '11pt' or '12pt').
+     #
+     # 'pointsize': '10pt',
+
+     # Additional stuff for the LaTeX preamble.
+     #
+     # 'preamble': '',
+
+     # Latex figure (float) alignment
+     #
+     # 'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+#  author, documentclass [howto, manual, or own class]).
+latex_documents = [
+    (master_doc, 'OPNFVReporting.tex',
+     u'OPNFV testing Reporting Documentation',
+     u'\\#opnfv-testperf (chat.freenode.net)', 'manual'),
+]
+
+# The name of an image file (relative to this directory) to place at the top of
+# the title page.
+#
+# latex_logo = None
+
+# For "manual" documents, if this is true, then toplevel headings are parts,
+# not chapters.
+#
+# latex_use_parts = False
+
+# If true, show page references after internal links.
+#
+# latex_show_pagerefs = False
+
+# If true, show URL addresses after external links.
+#
+# latex_show_urls = False
+
+# Documents to append as an appendix to all manuals.
+#
+# latex_appendices = []
+
+# It false, will not define \strong, \code,    itleref, \crossref ... but only
+# \sphinxstrong, ..., \sphinxtitleref, ... To help avoid clash with user added
+# packages.
+#
+# latex_keep_old_macro_names = True
+
+# If false, no module index is generated.
+#
+# latex_domain_indices = True
+
+
+# -- Options for manual page output ---------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+    (master_doc, 'opnfvReporting', u'OPNFV Testing Reporting Documentation',
+     [author], 1)
+]
+
+# If true, show URL addresses after external links.
+#
+# man_show_urls = False
+
+
+# -- Options for Texinfo output -------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+    (master_doc, 'OPNFVReporting', u'OPNFV Testing reporting Documentation',
+     author, 'OPNFVTesting', 'One line description of project.',
+     'Miscellaneous'),
+]
+
+# Documents to append as an appendix to all manuals.
+#
+# texinfo_appendices = []
+
+# If false, no module index is generated.
+#
+# texinfo_domain_indices = True
+
+# How to display URL addresses: 'footnote', 'no', or 'inline'.
+#
+# texinfo_show_urls = 'footnote'
+
+# If true, do not generate a @detailmenu in the "Top" node's menu.
+#
+# texinfo_no_detailmenu = False
diff --git a/utils/test/reporting/docs/index.rst b/utils/test/reporting/docs/index.rst
new file mode 100644 (file)
index 0000000..af41876
--- /dev/null
@@ -0,0 +1,16 @@
+Welcome to OPNFV Testing reporting documentation!
+=================================================
+
+Contents:
+
+.. toctree::
+   :maxdepth: 2
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
+
@@ -18,6 +18,8 @@ import scenarioResult as sr
 # manage conf
 import utils.reporting_utils as rp_utils
 
+"""Functest reporting status"""
+
 # Logger
 logger = rp_utils.getLogger("Functest-Status")
 
@@ -272,7 +274,8 @@ for version in versions:
             templateEnv = jinja2.Environment(
                 loader=templateLoader, autoescape=True)
 
-            TEMPLATE_FILE = "./functest/template/index-status-tmpl.html"
+            TEMPLATE_FILE = ("./reporting/functest/template"
+                             "/index-status-tmpl.html")
             template = templateEnv.get_template(TEMPLATE_FILE)
 
             outputText = template.render(
@@ -142,7 +142,7 @@ for version in rp_utils.get_config('general.versions'):
         templateEnv = jinja2.Environment(loader=templateLoader,
                                          autoescape=True)
 
-        TEMPLATE_FILE = "./functest/template/index-tempest-tmpl.html"
+        TEMPLATE_FILE = "./reporting/functest/template/index-tempest-tmpl.html"
         template = templateEnv.get_template(TEMPLATE_FILE)
 
         outputText = template.render(scenario_results=scenario_results,
@@ -104,7 +104,7 @@ for version in versions:
                                  % result['details']['sig_test']['duration'])
                     logger.debug("Signaling testing results: %s"
                                  % format_result)
-                except:
+                except Exception:
                     logger.error("Data badly formatted")
                 logger.debug("----------------------------------------")
 
@@ -112,7 +112,7 @@ for version in versions:
         templateEnv = jinja2.Environment(loader=templateLoader,
                                          autoescape=True)
 
-        TEMPLATE_FILE = "./functest/template/index-vims-tmpl.html"
+        TEMPLATE_FILE = "./reporting/functest/template/index-vims-tmpl.html"
         template = templateEnv.get_template(TEMPLATE_FILE)
 
         outputText = template.render(scenario_results=scenario_results,
@@ -23,7 +23,7 @@ reportingDate = datetime.datetime.now().strftime("%Y-%m-%d %H:%M")
 
 logger.info("*******************************************")
 logger.info("*   Generating reporting scenario status  *")
-logger.info("*   Data retention = %s days              *" % PERIOD)
+logger.info("*   Data retention = {} days              *".format(PERIOD))
 logger.info("*                                         *")
 logger.info("*******************************************")
 
@@ -33,7 +33,8 @@ def prepare_profile_file(version):
     if not os.path.exists(profile_dir):
         os.makedirs(profile_dir)
 
-    profile_file = '{}/scenario_history.txt'.format(profile_dir, version)
+    profile_file = "{}/{}/scenario_history.txt".format(profile_dir,
+                                                       version)
     if not os.path.exists(profile_file):
         with open(profile_file, 'w') as f:
             info = 'date,scenario,installer,details,score\n'
@@ -77,7 +78,7 @@ def render_html(prof_results, installer, version):
     template_env = jinja2.Environment(loader=template_loader,
                                       autoescape=True)
 
-    template_file = "./qtip/template/index-status-tmpl.html"
+    template_file = "./reporting/qtip/template/index-status-tmpl.html"
     template = template_env.get_template(template_file)
 
     render_outcome = template.render(prof_results=prof_results,
@@ -106,5 +107,6 @@ def render_reporter():
         rp_utils.generate_csv(profile_file)
         logger.info("CSV generated...")
 
+
 if __name__ == '__main__':
     render_reporter()
@@ -131,7 +131,7 @@ for version in versions:
         templateEnv = jinja2.Environment(loader=templateLoader,
                                          autoescape=True)
 
-        TEMPLATE_FILE = "./storperf/template/index-status-tmpl.html"
+        TEMPLATE_FILE = "./reporting/storperf/template/index-status-tmpl.html"
         template = templateEnv.get_template(TEMPLATE_FILE)
 
         outputText = template.render(scenario_results=scenario_result_criteria,
diff --git a/utils/test/reporting/reporting/tests/unit/utils/__init__.py b/utils/test/reporting/reporting/tests/unit/utils/__init__.py
new file mode 100644 (file)
index 0000000..e69de29
@@ -10,7 +10,7 @@
 import logging
 import unittest
 
-from utils import reporting_utils
+from reporting.utils import reporting_utils
 
 
 class reportingUtilsTesting(unittest.TestCase):
@@ -20,10 +20,9 @@ class reportingUtilsTesting(unittest.TestCase):
     def setUp(self):
         self.test = reporting_utils
 
-    def test_getConfig(self):
-        self.assertEqual(self.test.get_config("general.period"), 10)
-# TODO
-# ...
+    def test_foo(self):
+        self.assertTrue(0 < 1)
+
 
 if __name__ == "__main__":
     unittest.main(verbosity=2)
diff --git a/utils/test/reporting/reporting/utils/__init__.py b/utils/test/reporting/reporting/utils/__init__.py
new file mode 100644 (file)
index 0000000..e69de29
@@ -95,7 +95,7 @@ def getApiResults(case, installer, scenario, version):
         k = response.read()
         results = json.loads(k)
     except URLError as e:
-        print('No kittez. Got an error code:', e)
+        print 'No kittez. Got an error code:'.format(e)
 
     return results
 
@@ -124,19 +124,21 @@ def getScenarios(case, installer, version):
         k = response.read()
         results = json.loads(k)
         test_results = results['results']
-
-        page = results['pagination']['total_pages']
-        if page > 1:
-            test_results = []
-            for i in range(1, page + 1):
-                url_page = url + "&page=" + str(i)
-                request = Request(url_page)
-                response = urlopen(request)
-                k = response.read()
-                results = json.loads(k)
-                test_results += results['results']
+        try:
+            page = results['pagination']['total_pages']
+            if page > 1:
+                test_results = []
+                for i in range(1, page + 1):
+                    url_page = url + "&page=" + str(i)
+                    request = Request(url_page)
+                    response = urlopen(request)
+                    k = response.read()
+                    results = json.loads(k)
+                    test_results += results['results']
+        except KeyError:
+            print ('No pagination detected')
     except URLError as err:
-        print('Got an error code:', err)
+        print 'Got an error code: {}'.format(err)
 
     if test_results is not None:
         test_results.reverse()
@@ -152,7 +154,7 @@ def getScenarios(case, installer, version):
             exclude_noha = get_config('functest.exclude_noha')
             if ((exclude_virtual_pod and "virtual" in r['pod_name']) or
                     (exclude_noha and "noha" in r['scenario'])):
-                print("exclude virtual pod results...")
+                print "exclude virtual pod results..."
             else:
                 scenario_results[r['scenario']].append(r)
 
@@ -183,7 +185,7 @@ def getScenarioStatus(installer, version):
         results = json.loads(k)
         test_results = results['results']
     except URLError as e:
-        print('Got an error code:', e)
+        print 'Got an error code: {}'.format(e)
 
     scenario_results = {}
     result_dict = {}
@@ -223,7 +225,7 @@ def getQtipResults(version, installer):
         response.close()
         results = json.loads(k)['results']
     except URLError as err:
-        print('Got an error code:', err)
+        print 'Got an error code: {}'.format(err)
 
     result_dict = {}
     if results:
@@ -245,7 +247,7 @@ def getNbtestOk(results):
                 if "PASS" in v:
                     nb_test_ok += 1
             except:
-                print("Cannot retrieve test status")
+                print "Cannot retrieve test status"
     return nb_test_ok
 
 
@@ -320,7 +322,7 @@ def getJenkinsUrl(build_tag):
                   "/" + str(build_id[0]))
         jenkins_url = url_base + url_id + "/console"
     except:
-        print('Impossible to get jenkins url:')
+        print 'Impossible to get jenkins url:'
 
     if "jenkins-" not in build_tag:
         jenkins_url = None
@@ -333,7 +335,7 @@ def getScenarioPercent(scenario_score, scenario_criteria):
     try:
         score = float(scenario_score) / float(scenario_criteria) * 100
     except:
-        print('Impossible to calculate the percentage score')
+        print 'Impossible to calculate the percentage score'
     return score
 
 
@@ -419,7 +421,7 @@ def get_percent(four_list, ten_list):
 
 def _test():
     status = getScenarioStatus("compass", "master")
-    print("status:++++++++++++++++++++++++")
+    print "status:++++++++++++++++++++++++"
     print(json.dumps(status, indent=4))
 
 
@@ -456,6 +458,6 @@ def export_pdf(pdf_path, pdf_doc_name):
     try:
         pdfkit.from_file(pdf_path, pdf_doc_name)
     except IOError:
-        print("Error but pdf generated anyway...")
+        print "Error but pdf generated anyway..."
     except:
-        print("impossible to generate PDF")
+        print "impossible to generate PDF"
diff --git a/utils/test/reporting/reporting/yardstick/__init__.py b/utils/test/reporting/reporting/yardstick/__init__.py
new file mode 100644 (file)
index 0000000..e69de29
@@ -106,7 +106,7 @@ for version in versions:
         templateEnv = jinja2.Environment(loader=templateLoader,
                                          autoescape=True)
 
-        TEMPLATE_FILE = "./yardstick/template/index-status-tmpl.html"
+        TEMPLATE_FILE = "./reporting/yardstick/template/index-status-tmpl.html"
         template = templateEnv.get_template(TEMPLATE_FILE)
 
         outputText = template.render(scenario_results=scenario_result_criteria,
diff --git a/utils/test/reporting/requirements.txt b/utils/test/reporting/requirements.txt
new file mode 100644 (file)
index 0000000..344064d
--- /dev/null
@@ -0,0 +1,7 @@
+pdfkit>=0.6.1 # MIT
+wkhtmltopdf-pack>=0.12.3 # MIT
+PyYAML>=3.10.0 # MIT
+simplejson>=2.2.0 # MIT
+Jinja2!=2.9.0,!=2.9.1,!=2.9.2,!=2.9.3,!=2.9.4,>=2.8 # BSD License (3 clause)
+requests!=2.12.2,>=2.10.0 # Apache-2.0
+tornado>=4.4.2 # Apache-2.0
index 8c674ce..b83b550 100755 (executable)
@@ -1,44 +1,3 @@
 #!/bin/bash
-set -o errexit
-set -o pipefail
-
-
-# Get script directory
-SCRIPTDIR=`dirname $0`
-
-# Creating virtual environment
-if [ ! -z $VIRTUAL_ENV ]; then
-    venv=$VIRTUAL_ENV
-else
-    venv=$SCRIPTDIR/.venv
-    virtualenv $venv
-fi
-
-source $venv/bin/activate
-
-export CONFIG_REPORTING_YAML=$SCRIPTDIR/reporting.yaml
-
-# ***************
-# Run unit tests
-# ***************
-echo "Running unit tests..."
-
-# install python packages
-easy_install -U setuptools
-easy_install -U pip
-pip install -r $SCRIPTDIR/docker/requirements.pip
-pip install -e $SCRIPTDIR
-
-python $SCRIPTDIR/setup.py develop
-
-# unit tests
-# TODO: remove cover-erase
-# To be deleted when all functest packages will be listed
-nosetests --with-xunit \
-         --cover-package=$SCRIPTDIR/utils \
-         --with-coverage \
-         --cover-xml \
-         $SCRIPTDIR/tests/unit
-rc=$?
-
-deactivate
+tox
+exit $?
diff --git a/utils/test/reporting/setup.cfg b/utils/test/reporting/setup.cfg
new file mode 100644 (file)
index 0000000..9543945
--- /dev/null
@@ -0,0 +1,12 @@
+[metadata]
+name = reporting
+version = 1
+home-page = https://wiki.opnfv.org/display/testing
+
+[files]
+packages =
+    reporting
+    api
+scripts =
+    docker/reporting.sh
+    docker/web_server.sh
index 627785e..a52d905 100644 (file)
@@ -1,22 +1,24 @@
-##############################################################################
+#!/usr/bin/env python
+
+# Copyright (c) 2017 Orange and others.
+#
 # All rights reserved. This program and the accompanying materials
 # are made available under the terms of the Apache License, Version 2.0
 # which accompanies this distribution, and is available at
 # http://www.apache.org/licenses/LICENSE-2.0
-##############################################################################
 
-from setuptools import setup, find_packages
+# pylint: disable=missing-docstring
+
+import setuptools
 
+# In python < 2.7.4, a lazy loading of package `pbr` will break
+# setuptools if some other modules registered functions in `atexit`.
+# solution from: http://bugs.python.org/issue15881#msg170215
+try:
+    import multiprocessing  # noqa
+except ImportError:
+    pass
 
-setup(
-    name="reporting",
-    version="master",
-    packages=find_packages(),
-    include_package_data=True,
-    package_data={
-    },
-    url="https://www.opnfv.org",
-    install_requires=["coverage==4.1",
-                      "mock==1.3.0",
-                      "nose==1.3.7"],
-)
+setuptools.setup(
+    setup_requires=['pbr>=1.8'],
+    pbr=True)
diff --git a/utils/test/reporting/test-requirements.txt b/utils/test/reporting/test-requirements.txt
new file mode 100644 (file)
index 0000000..738f508
--- /dev/null
@@ -0,0 +1,5 @@
+coverage>=4.0 # Apache-2.0
+mock>=2.0 # BSD
+nose # LGPL
+flake8<2.6.0,>=2.5.4 # MIT
+pylint==1.4.5 # GPLv2
diff --git a/utils/test/reporting/tox.ini b/utils/test/reporting/tox.ini
new file mode 100644 (file)
index 0000000..2df5030
--- /dev/null
@@ -0,0 +1,27 @@
+[tox]
+envlist = pep8,pylint,py27
+
+[testenv]
+usedevelop = True
+deps =
+  -r{toxinidir}/requirements.txt
+  -r{toxinidir}/test-requirements.txt
+commands = nosetests --with-xunit \
+  --with-coverage \
+  --cover-tests \
+  --cover-package=reporting \
+  --cover-xml \
+  --cover-html \
+  reporting/tests/unit
+
+[testenv:pep8]
+basepython = python2.7
+commands = flake8
+
+[testenv:pylint]
+basepython = python2.7
+whitelist_externals = bash
+commands =
+  bash -c "\
+  pylint --disable=locally-disabled reporting| \
+    tee pylint.out | sed -ne '/Raw metrics/,//p'"