1 .. This work is licensed under a Creative Commons Attribution 4.0 International
3 .. http://creativecommons.org/licenses/by/4.0
4 .. (c) OPNFV, Ericsson AB, Huawei Technologies Co.,Ltd and others.
11 Yardstick supports installation by Docker or directly in Ubuntu. The
12 installation procedure for Docker and direct installation are detailed in
15 To use Yardstick you should have access to an OpenStack environment, with at
16 least Nova, Neutron, Glance, Keystone and Heat installed.
18 The steps needed to run Yardstick are:
21 2. Load OpenStack environment variables.
22 3. Create Yardstick flavor.
23 4. Build a guest image and load it into the OpenStack environment.
24 5. Create the test configuration ``.yaml`` file and run the test case/suite.
30 The OPNFV deployment is out of the scope of this document and can be found in
31 `User Guide & Configuration Guide`_. The OPNFV platform is considered as the
32 System Under Test (SUT) in this document.
34 Several prerequisites are needed for Yardstick:
36 1. A Jumphost to run Yardstick on
37 2. A Docker daemon or a virtual environment installed on the Jumphost
38 3. A public/external network created on the SUT
39 4. Connectivity from the Jumphost to the SUT public/external network
41 .. note:: *Jumphost* refers to any server which meets the previous
42 requirements. Normally it is the same server from where the OPNFV
43 deployment has been triggered.
45 .. warning:: Connectivity from Jumphost is essential and it is of paramount
46 importance to make sure it is working before even considering to install
47 and run Yardstick. Make also sure you understand how your networking is
50 .. note:: If your Jumphost is operating behind a company http proxy and/or
51 Firewall, please first consult `Proxy Support`_ section which is towards the
52 end of this document. That section details some tips/tricks which *may* be of
53 help in a proxified environment.
56 Install Yardstick using Docker (first option) (**recommended**)
57 ---------------------------------------------------------------
59 Yardstick has a Docker image. It is recommended to use this Docker image to run
62 Prepare the Yardstick container
63 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
65 Install docker on your guest system with the following command, if not done
68 wget -qO- https://get.docker.com/ | sh
70 Pull the Yardstick Docker image (``opnfv/yardstick``) from the public dockerhub
71 registry under the OPNFV account in dockerhub_, with the following docker
74 sudo -EH docker pull opnfv/yardstick:stable
76 After pulling the Docker image, check that it is available with the
77 following docker command::
79 [yardsticker@jumphost ~]$ docker images
80 REPOSITORY TAG IMAGE ID CREATED SIZE
81 opnfv/yardstick stable a4501714757a 1 day ago 915.4 MB
83 Run the Docker image to get a Yardstick container::
85 docker run -itd --privileged -v /var/run/docker.sock:/var/run/docker.sock \
86 -p 8888:5000 --name yardstick opnfv/yardstick:stable
88 .. table:: Description of the parameters used with ``docker run`` command
90 ======================= ====================================================
92 ======================= ====================================================
93 -itd -i: interactive, Keep STDIN open even if not
95 -t: allocate a pseudo-TTY detached mode, in the
97 ======================= ====================================================
98 --privileged If you want to build ``yardstick-image`` in
99 Yardstick container, this parameter is needed
100 ======================= ====================================================
101 -p 8888:5000 Redirect the a host port (8888) to a container port
103 ======================= ====================================================
104 -v /var/run/docker.sock If you want to use yardstick env grafana/influxdb to
105 :/var/run/docker.sock create a grafana/influxdb container out of Yardstick
107 ======================= ====================================================
108 --name yardstick The name for this container
110 If the host is restarted
111 ^^^^^^^^^^^^^^^^^^^^^^^^
113 The yardstick container must be started if the host is rebooted::
115 docker start yardstick
117 Configure the Yardstick container environment
118 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
120 There are three ways to configure environments for running Yardstick, explained
121 in the following sections. Before that, access the Yardstick container::
123 docker exec -it yardstick /bin/bash
125 and then configure Yardstick environments in the Yardstick container.
127 Using the CLI command ``env prepare`` (first way) (**recommended**)
128 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
130 In the Yardstick container, the Yardstick repository is located in the
131 ``/home/opnfv/repos`` directory. Yardstick provides a CLI to prepare OpenStack
132 environment variables and create Yardstick flavor and guest images
135 yardstick env prepare
137 .. note:: Since Euphrates release, the above command will not be able to
138 automatically configure the ``/etc/yardstick/openstack.creds`` file. So before
139 running the above command, it is necessary to create the
140 ``/etc/yardstick/openstack.creds`` file and save OpenStack environment
141 variables into it manually. If you have the openstack credential file saved
142 outside the Yardstick Docker container, you can do this easily by mapping the
143 credential file into Yardstick container using::
145 '-v /path/to/credential_file:/etc/yardstick/openstack.creds'
147 when running the Yardstick container. For details of the required OpenStack
148 environment variables please refer to section `Export OpenStack environment
151 The ``env prepare`` command may take up to 6-8 minutes to finish building
152 yardstick-image and other environment preparation. Meanwhile if you wish to
153 monitor the env prepare process, you can enter the Yardstick container in a new
154 terminal window and execute the following command::
156 tail -f /var/log/yardstick/uwsgi.log
159 Manually exporting the env variables and initializing OpenStack (second way)
160 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
162 Export OpenStack environment variables
163 ######################################
165 Before running Yardstick it is necessary to export OpenStack environment
170 Environment variables in the ``openrc`` file have to include at least::
178 A sample ``openrc`` file may look like this::
180 export OS_PASSWORD=console
181 export OS_TENANT_NAME=admin
182 export OS_AUTH_URL=http://172.16.1.222:35357/v2.0
183 export OS_USERNAME=admin
184 export OS_VOLUME_API_VERSION=2
185 export EXTERNAL_NETWORK=net04_ext
188 Manual creation of Yardstick flavor and guest images
189 ####################################################
191 Before executing Yardstick test cases, make sure that Yardstick flavor and
192 guest image are available in OpenStack. Detailed steps about creating the
193 Yardstick flavor and building the Yardstick guest image can be found below.
195 Most of the sample test cases in Yardstick are using an OpenStack flavor called
196 ``yardstick-flavor`` which deviates from the OpenStack standard ``m1.tiny``
197 flavor by the disk size; instead of 1GB it has 3GB. Other parameters are the
198 same as in ``m1.tiny``.
200 Create ``yardstick-flavor``::
202 openstack flavor create --disk 3 --vcpus 1 --ram 512 --swap 100 \
205 Most of the sample test cases in Yardstick are using a guest image called
206 ``yardstick-image`` which deviates from an Ubuntu Cloud Server image
207 containing all the required tools to run test cases supported by Yardstick.
208 Yardstick has a tool for building this custom image. It is necessary to have
209 ``sudo`` rights to use this tool.
211 Also you may need install several additional packages to use this tool, by
212 follwing the commands below::
214 sudo -EH apt-get update && sudo -EH apt-get install -y qemu-utils kpartx
216 This image can be built using the following command in the directory where
217 Yardstick is installed::
219 export YARD_IMG_ARCH='amd64'
220 echo "Defaults env_keep += \'YARD_IMG_ARCH\'" | sudo tee --append \
221 /etc/sudoers > /dev/null
222 sudo -EH tools/yardstick-img-modify tools/ubuntu-server-cloudimg-modify.sh
224 .. warning:: Before building the guest image inside the Yardstick container,
225 make sure the container is granted with privilege. The script will create files
226 by default in ``/tmp/workspace/yardstick`` and the files will be owned by root.
228 The created image can be added to OpenStack using the OpenStack client or via
229 the OpenStack Dashboard::
231 openstack image create --disk-format qcow2 --container-format bare \
232 --public --file /tmp/workspace/yardstick/yardstick-image.img \
236 Some Yardstick test cases use a `Cirros 0.3.5`_ image and/or a `Ubuntu 16.04`_
237 image. Add Cirros and Ubuntu images to OpenStack::
239 openstack image create --disk-format qcow2 --container-format bare \
240 --public --file $cirros_image_file cirros-0.3.5
241 openstack image create --disk-format qcow2 --container-format bare \
242 --file $ubuntu_image_file Ubuntu-16.04
245 Automatic initialization of OpenStack (third way)
246 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
248 Similar to the second way, the first step is also to
249 `Export OpenStack environment variables`_. Then the following steps should be
252 Automatic creation of Yardstick flavor and guest images
253 #######################################################
255 Yardstick has a script for automatically creating Yardstick flavor and building
256 Yardstick guest images. This script is mainly used for CI and can be also used
257 in the local environment::
259 source $YARDSTICK_REPO_DIR/tests/ci/load_images.sh
262 The Yardstick container GUI
263 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
265 In Euphrates release, Yardstick implemented a GUI for Yardstick Docker
266 container. After booting up Yardstick container, you can visit the GUI at
267 ``<container_host_ip>:8888/gui/index.html``.
269 For usage of Yardstick GUI, please watch our demo video at
270 `Yardstick GUI demo`_.
272 .. note:: The Yardstick GUI is still in development, the GUI layout and
275 Delete the Yardstick container
276 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
278 If you want to uninstall Yardstick, just delete the Yardstick container::
280 sudo docker stop yardstick && docker rm yardstick
284 Install Yardstick directly in Ubuntu (second option)
285 ----------------------------------------------------
287 .. _install-framework:
289 Alternatively you can install Yardstick framework directly in Ubuntu or in an
290 Ubuntu Docker image. No matter which way you choose to install Yardstick, the
291 following installation steps are identical.
293 If you choose to use the Ubuntu Docker image, you can pull the Ubuntu
294 Docker image from Docker hub::
296 sudo -EH docker pull ubuntu:16.04
302 Prerequisite preparation::
304 sudo -EH apt-get update && sudo -EH apt-get install -y \
305 git python-setuptools python-pip
306 sudo -EH easy_install -U setuptools==30.0.0
307 sudo -EH pip install appdirs==1.4.0
308 sudo -EH pip install virtualenv
310 Download the source code and install Yardstick from it::
312 git clone https://gerrit.opnfv.org/gerrit/yardstick
313 export YARDSTICK_REPO_DIR=~/yardstick
315 sudo -EH ./install.sh
317 If the host is ever restarted, nginx and uwsgi need to be restarted::
319 service nginx restart
320 uwsgi -i /etc/yardstick/yardstick.ini
322 Configure the Yardstick environment (**Todo**)
323 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
325 For installing Yardstick directly in Ubuntu, the ``yardstick env`` command is
326 not available. You need to prepare OpenStack environment variables and create
327 Yardstick flavor and guest images manually.
333 For uninstalling Yardstick, just delete the virtual environment::
335 rm -rf ~/yardstick_venv
338 Install Yardstick directly in OpenSUSE
339 --------------------------------------
341 .. _install-framework:
343 You can install Yardstick framework directly in OpenSUSE.
349 Prerequisite preparation::
351 sudo -EH zypper -n install -y gcc \
364 python-setuptools-git
366 Create a virtual environment::
368 virtualenv ~/yardstick_venv
369 export YARDSTICK_VENV=~/yardstick_venv
370 source ~/yardstick_venv/bin/activate
371 sudo -EH easy_install -U setuptools
373 Download the source code and install Yardstick from it::
375 git clone https://gerrit.opnfv.org/gerrit/yardstick
376 export YARDSTICK_REPO_DIR=~/yardstick
378 sudo -EH python setup.py install
379 sudo -EH pip install -r requirements.txt
381 Install missing python modules::
383 sudo -EH pip install pyyaml \
390 python.glanceclient \
391 python.neutronclient \
396 Configure the Yardstick environment
397 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
399 Source the OpenStack environment variables::
401 source DEVSTACK_DIRECTORY/openrc
403 Export the Openstack external network. The default installation of Devstack
404 names the external network public::
406 export EXTERNAL_NETWORK=public
407 export OS_USERNAME=demo
409 Change the API version used by Yardstick to v2.0 (the devstack openrc sets it
412 export OS_AUTH_URL=http://PUBLIC_IP_ADDRESS:5000/v2.0
418 For unistalling Yardstick, just delete the virtual environment::
420 rm -rf ~/yardstick_venv
423 Verify the installation
424 -----------------------
426 It is recommended to verify that Yardstick was installed successfully
427 by executing some simple commands and test samples. Before executing Yardstick
428 test cases make sure ``yardstick-flavor`` and ``yardstick-image`` can be found
429 in OpenStack and the ``openrc`` file is sourced. Below is an example invocation
430 of Yardstick ``help`` command and ``ping.py`` test sample::
433 yardstick task start samples/ping.yaml
435 .. note:: The above commands could be run in both the Yardstick container and
438 Each testing tool supported by Yardstick has a sample configuration file.
439 These configuration files can be found in the ``samples`` directory.
441 Default location for the output is ``/tmp/yardstick.out``.
444 Deploy InfluxDB and Grafana using Docker
445 ----------------------------------------
447 Without InfluxDB, Yardstick stores results for running test case in the file
448 ``/tmp/yardstick.out``. However, it's inconvenient to retrieve and display
449 test results. So we will show how to use InfluxDB to store data and use
450 Grafana to display data in the following sections.
452 Automatic deployment of InfluxDB and Grafana containers (**recommended**)
453 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
455 Firstly, enter the Yardstick container::
457 sudo -EH docker exec -it yardstick /bin/bash
459 Secondly, create InfluxDB container and configure with the following command::
461 yardstick env influxdb
463 Thirdly, create and configure Grafana container::
465 yardstick env grafana
467 Then you can run a test case and visit http://host_ip:1948
468 (``admin``/``admin``) to see the results.
470 .. note:: Executing ``yardstick env`` command to deploy InfluxDB and Grafana
471 requires Jumphost's docker API version => 1.24. Run the following command to
472 check the docker API version on the Jumphost::
477 Manual deployment of InfluxDB and Grafana containers
478 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
480 You can also deploy influxDB and Grafana containers manually on the Jumphost.
481 The following sections show how to do.
485 sudo -EH docker pull tutum/influxdb
486 sudo -EH docker pull grafana/grafana
490 sudo -EH docker run -d --name influxdb \
491 -p 8083:8083 -p 8086:8086 --expose 8090 --expose 8099 \
493 docker exec -it influxdb bash
498 >CREATE USER root WITH PASSWORD 'root' WITH ALL PRIVILEGES
499 >CREATE DATABASE yardstick;
505 sudo -EH docker run -d --name grafana -p 1948:3000 grafana/grafana
507 Log on http://{YOUR_IP_HERE}:1948 using ``admin``/``admin`` and configure
508 database resource to be ``{YOUR_IP_HERE}:8086``.
510 .. image:: images/Grafana_config.png
512 :alt: Grafana data source configuration
514 Configure ``yardstick.conf``::
516 sudo -EH docker exec -it yardstick /bin/bash
517 sudo cp etc/yardstick/yardstick.conf.sample /etc/yardstick/yardstick.conf
518 sudo vi /etc/yardstick/yardstick.conf
520 Modify ``yardstick.conf``::
524 dispatcher = influxdb
526 [dispatcher_influxdb]
528 target = http://{YOUR_IP_HERE}:8086
533 Now you can run Yardstick test cases and store the results in influxDB.
536 Deploy InfluxDB and Grafana directly in Ubuntu (**Todo**)
537 ---------------------------------------------------------
546 ``yardstick testcase list``: This command line would list all test cases in
547 Yardstick. It would show like below::
549 +---------------------------------------------------------------------------------------
550 | Testcase Name | Description
551 +---------------------------------------------------------------------------------------
552 | opnfv_yardstick_tc001 | Measure network throughput using pktgen
553 | opnfv_yardstick_tc002 | measure network latency using ping
554 | opnfv_yardstick_tc005 | Measure Storage IOPS, throughput and latency using fio.
556 +---------------------------------------------------------------------------------------
559 Show a test case config file
560 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
562 Take opnfv_yardstick_tc002 for an example. This test case measure network
563 latency. You just need to type in ``yardstick testcase show
564 opnfv_yardstick_tc002``, and the console would show the config yaml of this
569 schema: "yardstick:task:0.1"
571 Yardstick TC002 config file;
572 measure network latency using ping;
574 {% set image = image or "cirros-0.3.5" %}
576 {% set provider = provider or none %}
577 {% set physical_network = physical_network or 'physnet1' %}
578 {% set segmentation_id = segmentation_id or none %}
579 {% set packetsize = packetsize or 100 %}
582 {% for i in range(2) %}
586 packetsize: {{packetsize}}
603 flavor: yardstick-flavor
608 policy: "availability"
620 {% if provider == "vlan" %}
621 provider: {{provider}}
622 physical_network: {{physical_network}}å
623 {% if segmentation_id %}
624 segmentation_id: {{segmentation_id}}
629 Start a task to run yardstick test case
630 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
632 If you want run a test case, then you need to use ``yardstick task start
633 <test_case_path>`` this command support some parameters as below::
635 +---------------------+--------------------------------------------------+
636 | Parameters | Detail |
637 +=====================+==================================================+
638 | -d | show debug log of yardstick running |
640 +---------------------+--------------------------------------------------+
641 | --task-args | If you want to customize test case parameters, |
642 | | use "--task-args" to pass the value. The format |
643 | | is a json string with parameter key-value pair. |
645 +---------------------+--------------------------------------------------+
646 | --task-args-file | If you want to use yardstick |
647 | | env prepare command(or |
648 | | related API) to load the |
649 +---------------------+--------------------------------------------------+
653 +---------------------+--------------------------------------------------+
654 | --output-file \ | Specify where to output the log. if not pass, |
655 | OUTPUT_FILE_PATH | the default value is |
656 | | "/tmp/yardstick/yardstick.log" |
658 +---------------------+--------------------------------------------------+
659 | --suite \ | run a test suite, TEST_SUITE_PATH specify where |
660 | TEST_SUITE_PATH | the test suite locates |
662 +---------------------+--------------------------------------------------+
665 Run Yardstick in a local environment
666 ------------------------------------
668 We also have a guide about how to run Yardstick in a local environment.
669 This work is contributed by Tapio Tallgren.
670 You can find this guide at `How to run Yardstick in a local environment`_.
673 Create a test suite for Yardstick
674 ------------------------------------
676 A test suite in yardstick is a yaml file which include one or more test cases.
677 Yardstick is able to support running test suite task, so you can customize your
678 own test suite and run it in one task.
680 ``tests/opnfv/test_suites`` is the folder where Yardstick puts CI test suite.
681 A typical test suite is like below (the ``fuel_test_suite.yaml`` example)::
684 # Fuel integration test task suite
686 schema: "yardstick:suite:0.1"
688 name: "fuel_test_suite"
689 test_cases_dir: "samples/"
694 file_name: iperf3.yaml
696 As you can see, there are two test cases in the ``fuel_test_suite.yaml``. The
697 ``schema`` and the ``name`` must be specified. The test cases should be listed
698 via the tag ``test_cases`` and their relative path is also marked via the tag
701 Yardstick test suite also supports constraints and task args for each test
702 case. Here is another sample (the ``os-nosdn-nofeature-ha.yaml`` example) to
703 show this, which is digested from one big test suite::
707 schema: "yardstick:suite:0.1"
709 name: "os-nosdn-nofeature-ha"
710 test_cases_dir: "tests/opnfv/test_cases/"
713 file_name: opnfv_yardstick_tc002.yaml
715 file_name: opnfv_yardstick_tc005.yaml
717 file_name: opnfv_yardstick_tc043.yaml
722 huawei-pod1: '{"pod_info": "etc/yardstick/.../pod.yaml",
723 "host": "node4.LF","target": "node5.LF"}'
725 As you can see in test case ``opnfv_yardstick_tc043.yaml``, there are two
726 tags, ``constraint`` and ``task_args``. ``constraint`` is to specify which
727 installer or pod it can be run in the CI environment. ``task_args`` is to
728 specify the task arguments for each pod.
730 All in all, to create a test suite in Yardstick, you just need to create a
731 yaml file and add test cases, constraint or task arguments if necessary.
737 To configure the Jumphost to access Internet through a proxy its necessary to
738 export several variables to the environment, contained in the following
742 _proxy=<proxy_address>
743 _proxyport=<proxy_port>
744 _ip=$(hostname -I | awk '{print $1}')
746 export ftp_proxy=http://$_proxy:$_proxyport
747 export FTP_PROXY=http://$_proxy:$_proxyport
748 export http_proxy=http://$_proxy:$_proxyport
749 export HTTP_PROXY=http://$_proxy:$_proxyport
750 export https_proxy=http://$_proxy:$_proxyport
751 export HTTPS_PROXY=http://$_proxy:$_proxyport
752 export no_proxy=127.0.0.1,localhost,$_ip,$(hostname),<.localdomain>
753 export NO_PROXY=127.0.0.1,localhost,$_ip,$(hostname),<.localdomain>
755 To enable Internet access from a container using ``docker``, depends on the OS
756 version. On Ubuntu 14.04 LTS, which uses SysVinit, ``/etc/default/docker`` must
760 # If you need Docker to use an HTTP proxy, it can also be specified here.
761 export http_proxy="http://<proxy_address>:<proxy_port>/"
762 export https_proxy="https://<proxy_address>:<proxy_port>/"
764 Then its necessary to restart the ``docker`` service::
766 sudo -EH service docker restart
768 In Ubuntu 16.04 LTS, which uses Systemd, its necessary to create a drop-in
771 sudo mkdir /etc/systemd/system/docker.service.d
773 Then, the proxy configuration will be stored in the following file::
775 # cat /etc/systemd/system/docker.service.d/http-proxy.conf
777 Environment="HTTP_PROXY=https://<proxy_address>:<proxy_port>/"
778 Environment="HTTPS_PROXY=https://<proxy_address>:<proxy_port>/"
779 Environment="NO_PROXY=localhost,127.0.0.1,<localaddress>,<.localdomain>"
781 The changes need to be flushed and the ``docker`` service restarted::
783 sudo systemctl daemon-reload
784 sudo systemctl restart docker
786 Any container is already created won't contain these modifications. If needed,
787 stop and delete the container::
789 sudo docker stop yardstick
790 sudo docker rm yardstick
792 .. warning:: Be careful, the above ``rm`` command will delete the container
793 completely. Everything on this container will be lost.
795 Then follow the previous instructions `Prepare the Yardstick container`_ to
796 rebuild the Yardstick container.
802 .. _`User Guide & Configuration Guide`: http://docs.opnfv.org/en/latest/release/userguide.introduction.html
803 .. _dockerhub: https://hub.docker.com/r/opnfv/yardstick/
804 .. _`Cirros 0.3.5`: http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img
805 .. _`Ubuntu 16.04`: https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img
806 .. _`Yardstick GUI demo`: https://www.youtube.com/watch?v=M3qbJDp6QBk
807 .. _`How to run Yardstick in a local environment`: https://wiki.opnfv.org/display/yardstick/How+to+run+Yardstick+in+a+local+environment