1 .. This work is licensed under a Creative Commons Attribution 4.0 International
3 http://creativecommons.org/licenses/by/4.0
4 (c) OPNFV, Ericsson AB, Huawei Technologies Co.,Ltd and others.
7 Convention for heading levels in Yardstick documentation:
8 ======= Heading 0 (reserved for the title in a document)
14 Avoid deeper levels because they do not render well.
17 Convention for heading levels in Yardstick documentation:
19 ======= Heading 0 (reserved for the title in a document)
25 Avoid deeper levels because they do not render well.
27 ======================
28 Yardstick Installation
29 ======================
31 Yardstick supports installation by Docker or directly in Ubuntu. The
32 installation procedure for Docker and direct installation are detailed in
35 To use Yardstick you should have access to an OpenStack environment, with at
36 least Nova, Neutron, Glance, Keystone and Heat installed.
38 The steps needed to run Yardstick are:
41 2. Load OpenStack environment variables.
42 3. Create Yardstick flavor.
43 4. Build a guest image and load it into the OpenStack environment.
44 5. Create the test configuration ``.yaml`` file and run the test case/suite.
50 The OPNFV deployment is out of the scope of this document and can be found in
51 `User Guide & Configuration Guide`_. The OPNFV platform is considered as the
52 System Under Test (SUT) in this document.
54 Several prerequisites are needed for Yardstick:
56 1. A Jumphost to run Yardstick on
57 2. A Docker daemon or a virtual environment installed on the Jumphost
58 3. A public/external network created on the SUT
59 4. Connectivity from the Jumphost to the SUT public/external network
61 .. note:: *Jumphost* refers to any server which meets the previous
62 requirements. Normally it is the same server from where the OPNFV
63 deployment has been triggered.
65 .. warning:: Connectivity from Jumphost is essential and it is of paramount
66 importance to make sure it is working before even considering to install
67 and run Yardstick. Make also sure you understand how your networking is
70 .. note:: If your Jumphost is operating behind a company http proxy and/or
71 Firewall, please first consult `Proxy Support`_ section which is towards
72 the end of this document. That section details some tips/tricks which *may*
73 be of help in a proxified environment.
75 .. _Install Yardstick using Docker:
77 Install Yardstick using Docker (first option) (**recommended**)
78 ---------------------------------------------------------------
80 Yardstick has a Docker image. It is recommended to use this Docker image to run
83 Prepare the Yardstick container
84 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
86 Install docker on your guest system with the following command, if not done
89 wget -qO- https://get.docker.com/ | sh
91 Pull the Yardstick Docker image (``opnfv/yardstick``) from the public dockerhub
92 registry under the OPNFV account in dockerhub_, with the following docker
95 sudo -EH docker pull opnfv/yardstick:stable
97 After pulling the Docker image, check that it is available with the
98 following docker command::
100 [yardsticker@jumphost ~]$ docker images
101 REPOSITORY TAG IMAGE ID CREATED SIZE
102 opnfv/yardstick stable a4501714757a 1 day ago 915.4 MB
104 Run the Docker image to get a Yardstick container::
106 docker run -itd --privileged -v /var/run/docker.sock:/var/run/docker.sock \
107 -p 8888:5000 --name yardstick opnfv/yardstick:stable
109 Description of the parameters used with ``docker run`` command
111 +------------------------+--------------------------------------------------+
112 | Parameters | Detail |
113 +========================+==================================================+
114 | -itd | -i: interactive, Keep STDIN open even if not |
116 | +--------------------------------------------------+
117 | | -t: allocate a pseudo-TTY detached mode, in the |
119 +------------------------+--------------------------------------------------+
120 | --privileged | If you want to build ``yardstick-image`` in |
121 | | Yardstick container, this parameter is needed |
122 +------------------------+--------------------------------------------------+
123 | -p 8888:5000 | Redirect the a host port (8888) to a container |
125 +------------------------+--------------------------------------------------+
126 | -v /var/run/docker.sock| If you want to use yardstick env |
127 | :/var/run/docker.sock | grafana/influxdb to create a grafana/influxdb |
128 | | container out of Yardstick container |
129 +------------------------+--------------------------------------------------+
130 | --name yardstick | The name for this container |
131 +------------------------+--------------------------------------------------+
134 If the host is restarted
135 ^^^^^^^^^^^^^^^^^^^^^^^^
137 The yardstick container must be started if the host is rebooted::
139 docker start yardstick
141 Configure the Yardstick container environment
142 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
144 There are three ways to configure environments for running Yardstick, explained
145 in the following sections. Before that, access the Yardstick container::
147 docker exec -it yardstick /bin/bash
149 and then configure Yardstick environments in the Yardstick container.
151 Using the CLI command ``env prepare`` (first way) (**recommended**)
152 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
154 In the Yardstick container, the Yardstick repository is located in the
155 ``/home/opnfv/repos`` directory. Yardstick provides a CLI to prepare OpenStack
156 environment variables and create Yardstick flavor and guest images
159 yardstick env prepare
161 .. note:: Since Euphrates release, the above command will not be able to
162 automatically configure the ``/etc/yardstick/openstack.creds`` file. So before
163 running the above command, it is necessary to create the
164 ``/etc/yardstick/openstack.creds`` file and save OpenStack environment
165 variables into it manually. If you have the openstack credential file saved
166 outside the Yardstick Docker container, you can do this easily by mapping the
167 credential file into Yardstick container using::
169 '-v /path/to/credential_file:/etc/yardstick/openstack.creds'
171 when running the Yardstick container. For details of the required OpenStack
172 environment variables please refer to section `Export OpenStack environment
175 The ``env prepare`` command may take up to 6-8 minutes to finish building
176 yardstick-image and other environment preparation. Meanwhile if you wish to
177 monitor the env prepare process, you can enter the Yardstick container in a new
178 terminal window and execute the following command::
180 tail -f /var/log/yardstick/uwsgi.log
183 Manually exporting the env variables and initializing OpenStack (second way)
184 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
186 Export OpenStack environment variables
187 ''''''''''''''''''''''''''''''''''''''
189 Before running Yardstick it is necessary to export OpenStack environment
194 Environment variables in the ``openrc`` file have to include at least::
202 A sample ``openrc`` file may look like this::
204 export OS_PASSWORD=console
205 export OS_PROJECT_NAME=admin
206 export OS_AUTH_URL=http://172.16.1.222:35357/v2.0
207 export OS_USERNAME=admin
208 export OS_VOLUME_API_VERSION=2
209 export EXTERNAL_NETWORK=net04_ext
212 Manual creation of Yardstick flavor and guest images
213 ''''''''''''''''''''''''''''''''''''''''''''''''''''
215 Before executing Yardstick test cases, make sure that Yardstick flavor and
216 guest image are available in OpenStack. Detailed steps about creating the
217 Yardstick flavor and building the Yardstick guest image can be found below.
219 Most of the sample test cases in Yardstick are using an OpenStack flavor called
220 ``yardstick-flavor`` which deviates from the OpenStack standard ``m1.tiny``
221 flavor by the disk size; instead of 1GB it has 3GB. Other parameters are the
222 same as in ``m1.tiny``.
224 Create ``yardstick-flavor``::
226 openstack flavor create --disk 3 --vcpus 1 --ram 512 --swap 100 \
229 Most of the sample test cases in Yardstick are using a guest image called
230 ``yardstick-image`` which deviates from an Ubuntu Cloud Server image
231 containing all the required tools to run test cases supported by Yardstick.
232 Yardstick has a tool for building this custom image. It is necessary to have
233 ``sudo`` rights to use this tool.
235 Also you may need install several additional packages to use this tool, by
236 follwing the commands below::
238 sudo -EH apt-get update && sudo -EH apt-get install -y qemu-utils kpartx
240 This image can be built using the following command in the directory where
241 Yardstick is installed::
243 export YARD_IMG_ARCH='amd64'
244 echo "Defaults env_keep += \'YARD_IMG_ARCH\'" | sudo tee --append \
245 /etc/sudoers > /dev/null
246 sudo -EH tools/yardstick-img-modify tools/ubuntu-server-cloudimg-modify.sh
248 .. warning:: Before building the guest image inside the Yardstick container,
249 make sure the container is granted with privilege. The script will create files
250 by default in ``/tmp/workspace/yardstick`` and the files will be owned by root.
252 The created image can be added to OpenStack using the OpenStack client or via
253 the OpenStack Dashboard::
255 openstack image create --disk-format qcow2 --container-format bare \
256 --public --file /tmp/workspace/yardstick/yardstick-image.img \
260 Some Yardstick test cases use a `Cirros 0.3.5`_ image and/or a `Ubuntu 16.04`_
261 image. Add Cirros and Ubuntu images to OpenStack::
263 openstack image create --disk-format qcow2 --container-format bare \
264 --public --file $cirros_image_file cirros-0.3.5
265 openstack image create --disk-format qcow2 --container-format bare \
266 --file $ubuntu_image_file Ubuntu-16.04
269 Automatic initialization of OpenStack (third way)
270 +++++++++++++++++++++++++++++++++++++++++++++++++
272 Similar to the second way, the first step is also to
273 `Export OpenStack environment variables`_. Then the following steps should be
276 Automatic creation of Yardstick flavor and guest images
277 '''''''''''''''''''''''''''''''''''''''''''''''''''''''
279 Yardstick has a script for automatically creating Yardstick flavor and building
280 Yardstick guest images. This script is mainly used for CI and can be also used
281 in the local environment::
283 source $YARDSTICK_REPO_DIR/tests/ci/load_images.sh
286 The Yardstick container GUI
287 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
289 In Euphrates release, Yardstick implemented a GUI for Yardstick Docker
290 container. After booting up Yardstick container, you can visit the GUI at
291 ``<container_host_ip>:8888/gui/index.html``.
293 For usage of Yardstick GUI, please watch our demo video at
294 `Yardstick GUI demo`_.
296 .. note:: The Yardstick GUI is still in development, the GUI layout and
299 Delete the Yardstick container
300 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
302 If you want to uninstall Yardstick, just delete the Yardstick container::
304 sudo docker stop yardstick && docker rm yardstick
308 Install Yardstick directly in Ubuntu (second option)
309 ----------------------------------------------------
311 .. _install-framework:
313 Alternatively you can install Yardstick framework directly in Ubuntu or in an
314 Ubuntu Docker image. No matter which way you choose to install Yardstick, the
315 following installation steps are identical.
317 If you choose to use the Ubuntu Docker image, you can pull the Ubuntu
318 Docker image from Docker hub::
320 sudo -EH docker pull ubuntu:16.04
326 Prerequisite preparation::
328 sudo -EH apt-get update && sudo -EH apt-get install -y \
329 git python-setuptools python-pip
330 sudo -EH easy_install -U setuptools==30.0.0
331 sudo -EH pip install appdirs==1.4.0
332 sudo -EH pip install virtualenv
334 Download the source code and install Yardstick from it::
336 git clone https://gerrit.opnfv.org/gerrit/yardstick
337 export YARDSTICK_REPO_DIR=~/yardstick
339 sudo -EH ./install.sh
341 If the host is ever restarted, nginx and uwsgi need to be restarted::
343 service nginx restart
344 uwsgi -i /etc/yardstick/yardstick.ini
346 Configure the Yardstick environment (**Todo**)
347 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
349 For installing Yardstick directly in Ubuntu, the ``yardstick env`` command is
350 not available. You need to prepare OpenStack environment variables and create
351 Yardstick flavor and guest images manually.
356 For uninstalling Yardstick, just delete the virtual environment::
358 rm -rf ~/yardstick_venv
361 Install Yardstick directly in OpenSUSE
362 --------------------------------------
364 .. _install-framework:
366 You can install Yardstick framework directly in OpenSUSE.
372 Prerequisite preparation::
374 sudo -EH zypper -n install -y gcc \
387 python-setuptools-git
389 Create a virtual environment::
391 virtualenv ~/yardstick_venv
392 export YARDSTICK_VENV=~/yardstick_venv
393 source ~/yardstick_venv/bin/activate
394 sudo -EH easy_install -U setuptools
396 Download the source code and install Yardstick from it::
398 git clone https://gerrit.opnfv.org/gerrit/yardstick
399 export YARDSTICK_REPO_DIR=~/yardstick
401 sudo -EH python setup.py install
402 sudo -EH pip install -r requirements.txt
404 Install missing python modules::
406 sudo -EH pip install pyyaml \
413 python.glanceclient \
414 python.neutronclient \
419 Configure the Yardstick environment
420 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
422 Source the OpenStack environment variables::
424 source DEVSTACK_DIRECTORY/openrc
426 Export the Openstack external network. The default installation of Devstack
427 names the external network public::
429 export EXTERNAL_NETWORK=public
430 export OS_USERNAME=demo
432 Change the API version used by Yardstick to v2.0 (the devstack openrc sets it
435 export OS_AUTH_URL=http://PUBLIC_IP_ADDRESS:5000/v2.0
441 For unistalling Yardstick, just delete the virtual environment::
443 rm -rf ~/yardstick_venv
446 Verify the installation
447 -----------------------
449 It is recommended to verify that Yardstick was installed successfully
450 by executing some simple commands and test samples. Before executing Yardstick
451 test cases make sure ``yardstick-flavor`` and ``yardstick-image`` can be found
452 in OpenStack and the ``openrc`` file is sourced. Below is an example invocation
453 of Yardstick ``help`` command and ``ping.py`` test sample::
456 yardstick task start samples/ping.yaml
458 .. note:: The above commands could be run in both the Yardstick container and
461 Each testing tool supported by Yardstick has a sample configuration file.
462 These configuration files can be found in the ``samples`` directory.
464 Default location for the output is ``/tmp/yardstick.out``.
467 Automatic installation of Yardstick
468 -----------------------------------
470 Automatic installation can be used as an alternative to the manual by
471 providing parameters for ansible script ``install.yaml`` in a ``nsb_setup.sh``
472 file. Yardstick can be installed on the bare metal and to the container. Yardstick
473 container can be either pulled or built.
475 Bare metal installation
476 ^^^^^^^^^^^^^^^^^^^^^^^
478 Modify ``nsb_setup.sh`` file ``install.yaml`` parameters to install Yardstick
481 .. code-block:: console
483 ansible-playbook -i install-inventory.ini install.yaml \
484 -e IMAGE_PROPERTY='none' \
485 -e YARDSTICK_DIR=<path to Yardstick folder>
487 .. note:: By default ``INSTALLATION_MODE`` is ``baremetal``.
489 .. note:: No modification in ``install-inventory.ini`` is needed for Yardstick
492 .. note:: To install Yardstick in virtual environment pass parameter
493 ``-e VIRTUAL_ENVIRONMENT=True``.
495 Container installation
496 ^^^^^^^^^^^^^^^^^^^^^^
498 Modify ``install.yaml`` parameters in ``nsb_setup.sh`` file to pull or build
499 Yardstick container. To pull Yardstick image and start container run:
501 .. code-block:: console
503 ansible-playbook -i install-inventory.ini install.yaml \
504 -e IMAGE_PROPERTY='none' \
505 -e INSTALLATION_MODE=container_pull
507 .. note:: Yardstick docker image is available for both Ubuntu 16.04 and Ubuntu
508 18.04. By default Ubuntu 16.04 based docker image is used. To use
509 Ubuntu 18.04 based docker image pass ``-i opnfv/yardstick-ubuntu-18.04``
510 parameter to ``nsb_setup.sh``.
512 To build Yardstick image modify Dockerfile as per comments in it and run:
514 .. code-block:: console
517 docker build -f docker/Dockerfile -t opnfv/yardstick:<tag> .
519 .. note:: Yardstick docker image based on Ubuntu 16.04 will be built.
520 Pass ``-f docker/Dockerfile_ubuntu18`` to build Yardstick docker image based
523 .. note:: Add ``--build-arg http_proxy=http://<proxy_host>:<proxy_port>`` to
524 build docker image if server is behind the proxy.
526 Parameters for ``install.yaml``
527 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
529 Description of the parameters used with ``install.yaml``:
531 +-------------------------+-------------------------------------------------+
532 | Parameters | Detail |
533 +=========================+=================================================+
534 | -i install-inventory.ini|| Installs package dependency to remote servers |
536 | || Mandatory parameter |
537 | || By default no remote servers are provided |
538 +-------------------------+-------------------------------------------------+
539 | -e YARDSTICK_DIR || Path to Yardstick folder |
540 | || Mandatory parameter for Yardstick bare metal |
542 +-------------------------+-------------------------------------------------+
543 | -e INSTALLATION_MODE || baremetal: Yardstick is installed to the bare |
545 | || Default parameter |
546 | +-------------------------------------------------+
547 | || container: Yardstick is installed in container |
548 | || Container is built from Dockerfile |
549 | +-------------------------------------------------+
550 | || container_pull: Yardstick is installed in |
552 | || Container is pulled from docker hub |
553 +-------------------------+-------------------------------------------------+
554 | -e OS_RELEASE || xenial or bionic: Ubuntu version to be used for|
555 | || VM image (nsb or normal) |
556 | || Default is Ubuntu 16.04, xenial |
557 +-------------------------+-------------------------------------------------+
558 | -e IMAGE_PROPERTY || nsb: Build Yardstick NSB VM image |
559 | || Used to run Yardstick NSB tests on sample VNF |
560 | || Default parameter |
561 | +-------------------------------------------------+
562 | || normal: Build VM image to run ping test in |
564 | +-------------------------------------------------+
565 | || none: don't build a VM image. |
566 +-------------------------+-------------------------------------------------+
567 | -e VIRTUAL_ENVIRONMENT || False or True: Whether install in virtualenv |
568 | || Default is False |
569 +-------------------------+-------------------------------------------------+
570 | -e YARD_IMAGE_ARCH || CPU architecture on servers |
571 | || Default is 'amd64' |
572 +-------------------------+-------------------------------------------------+
575 Deploy InfluxDB and Grafana using Docker
576 ----------------------------------------
578 Without InfluxDB, Yardstick stores results for running test case in the file
579 ``/tmp/yardstick.out``. However, it's inconvenient to retrieve and display
580 test results. So we will show how to use InfluxDB to store data and use
581 Grafana to display data in the following sections.
583 Automatic deployment of InfluxDB and Grafana containers (**recommended**)
584 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
586 1. Enter the Yardstick container::
588 sudo -EH docker exec -it yardstick /bin/bash
590 2. Create InfluxDB container and configure with the following command::
592 yardstick env influxdb
594 3. Create and configure Grafana container::
596 yardstick env grafana
598 Then you can run a test case and visit http://host_ip:1948
599 (``admin``/``admin``) to see the results.
601 .. note:: Executing ``yardstick env`` command to deploy InfluxDB and Grafana
602 requires Jumphost's docker API version => 1.24. Run the following command to
603 check the docker API version on the Jumphost::
608 Manual deployment of InfluxDB and Grafana containers
609 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
611 You can also deploy influxDB and Grafana containers manually on the Jumphost.
612 The following sections show how to do.
616 sudo -EH docker pull tutum/influxdb
617 sudo -EH docker pull grafana/grafana
621 sudo -EH docker run -d --name influxdb \
622 -p 8083:8083 -p 8086:8086 --expose 8090 --expose 8099 \
627 docker exec -it influxdb influx
628 > CREATE USER root WITH PASSWORD 'root' WITH ALL PRIVILEGES
629 > CREATE DATABASE yardstick;
636 sudo -EH docker run -d --name grafana -p 1948:3000 grafana/grafana
638 Log on to ``http://{YOUR_IP_HERE}:1948`` using ``admin``/``admin`` and configure
639 database resource to be ``{YOUR_IP_HERE}:8086``.
641 .. image:: images/Grafana_config.png
643 :alt: Grafana data source configuration
645 Configure ``yardstick.conf``::
647 sudo -EH docker exec -it yardstick /bin/bash
648 sudo cp etc/yardstick/yardstick.conf.sample /etc/yardstick/yardstick.conf
649 sudo vi /etc/yardstick/yardstick.conf
651 Modify ``yardstick.conf`` to add the ``influxdb`` dispatcher::
655 dispatcher = influxdb
657 [dispatcher_influxdb]
659 target = http://{YOUR_IP_HERE}:8086
664 Now Yardstick will store results in InfluxDB when you run a testcase.
666 Deploy InfluxDB and Grafana directly in Ubuntu (**Todo**)
667 ---------------------------------------------------------
672 To configure the Jumphost to access Internet through a proxy its necessary to
673 export several variables to the environment, contained in the following
677 _proxy=<proxy_address>
678 _proxyport=<proxy_port>
679 _ip=$(hostname -I | awk '{print $1}')
681 export ftp_proxy=http://$_proxy:$_proxyport
682 export FTP_PROXY=http://$_proxy:$_proxyport
683 export http_proxy=http://$_proxy:$_proxyport
684 export HTTP_PROXY=http://$_proxy:$_proxyport
685 export https_proxy=http://$_proxy:$_proxyport
686 export HTTPS_PROXY=http://$_proxy:$_proxyport
687 export no_proxy=127.0.0.1,localhost,$_ip,$(hostname),<.localdomain>
688 export NO_PROXY=127.0.0.1,localhost,$_ip,$(hostname),<.localdomain>
690 To enable Internet access from a container using ``docker``, depends on the OS
691 version. On Ubuntu 14.04 LTS, which uses SysVinit, ``/etc/default/docker`` must
695 # If you need Docker to use an HTTP proxy, it can also be specified here.
696 export http_proxy="http://<proxy_address>:<proxy_port>/"
697 export https_proxy="https://<proxy_address>:<proxy_port>/"
699 Then its necessary to restart the ``docker`` service::
701 sudo -EH service docker restart
703 In Ubuntu 16.04 LTS, which uses Systemd, its necessary to create a drop-in
706 sudo mkdir /etc/systemd/system/docker.service.d
708 Then, the proxy configuration will be stored in the following file::
710 # cat /etc/systemd/system/docker.service.d/http-proxy.conf
712 Environment="HTTP_PROXY=https://<proxy_address>:<proxy_port>/"
713 Environment="HTTPS_PROXY=https://<proxy_address>:<proxy_port>/"
714 Environment="NO_PROXY=localhost,127.0.0.1,<localaddress>,<.localdomain>"
716 The changes need to be flushed and the ``docker`` service restarted::
718 sudo systemctl daemon-reload
719 sudo systemctl restart docker
721 Any container is already created won't contain these modifications. If needed,
722 stop and delete the container::
724 sudo docker stop yardstick
725 sudo docker rm yardstick
727 .. warning:: Be careful, the above ``rm`` command will delete the container
728 completely. Everything on this container will be lost.
730 Then follow the previous instructions `Prepare the Yardstick container`_ to
731 rebuild the Yardstick container.
737 .. _`User Guide & Configuration Guide`: http://docs.opnfv.org/en/latest/release/userguide.introduction.html
738 .. _dockerhub: https://hub.docker.com/r/opnfv/yardstick/
739 .. _`Cirros 0.3.5`: http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img
740 .. _`Ubuntu 16.04`: https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img
741 .. _`Yardstick GUI demo`: https://www.youtube.com/watch?v=M3qbJDp6QBk