1 .. This work is licensed under a Creative Commons Attribution 4.0 International
3 .. http://creativecommons.org/licenses/by/4.0
4 .. (c) OPNFV, Ericsson AB, Huawei Technologies Co.,Ltd and others.
11 Yardstick supports installation by Docker or directly in Ubuntu. The
12 installation procedure for Docker and direct installation are detailed in
15 To use Yardstick you should have access to an OpenStack environment, with at
16 least Nova, Neutron, Glance, Keystone and Heat installed.
18 The steps needed to run Yardstick are:
21 2. Load OpenStack environment variables.
22 3. Create Yardstick flavor.
23 4. Build a guest image and load it into the OpenStack environment.
24 5. Create the test configuration ``.yaml`` file and run the test case/suite.
30 The OPNFV deployment is out of the scope of this document and can be found in
31 `User Guide & Configuration Guide`_. The OPNFV platform is considered as the
32 System Under Test (SUT) in this document.
34 Several prerequisites are needed for Yardstick:
36 1. A Jumphost to run Yardstick on
37 2. A Docker daemon or a virtual environment installed on the Jumphost
38 3. A public/external network created on the SUT
39 4. Connectivity from the Jumphost to the SUT public/external network
41 .. note:: *Jumphost* refers to any server which meets the previous
42 requirements. Normally it is the same server from where the OPNFV
43 deployment has been triggered.
45 .. warning:: Connectivity from Jumphost is essential and it is of paramount
46 importance to make sure it is working before even considering to install
47 and run Yardstick. Make also sure you understand how your networking is
50 .. note:: If your Jumphost is operating behind a company http proxy and/or
51 Firewall, please first consult `Proxy Support`_ section which is towards
52 the end of this document. That section details some tips/tricks which *may*
53 be of help in a proxified environment.
56 Install Yardstick using Docker (first option) (**recommended**)
57 ---------------------------------------------------------------
59 Yardstick has a Docker image. It is recommended to use this Docker image to run
62 Prepare the Yardstick container
63 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
65 Install docker on your guest system with the following command, if not done
68 wget -qO- https://get.docker.com/ | sh
70 Pull the Yardstick Docker image (``opnfv/yardstick``) from the public dockerhub
71 registry under the OPNFV account in dockerhub_, with the following docker
74 sudo -EH docker pull opnfv/yardstick:stable
76 After pulling the Docker image, check that it is available with the
77 following docker command::
79 [yardsticker@jumphost ~]$ docker images
80 REPOSITORY TAG IMAGE ID CREATED SIZE
81 opnfv/yardstick stable a4501714757a 1 day ago 915.4 MB
83 Run the Docker image to get a Yardstick container::
85 docker run -itd --privileged -v /var/run/docker.sock:/var/run/docker.sock \
86 -p 8888:5000 --name yardstick opnfv/yardstick:stable
88 Description of the parameters used with ``docker run`` command
90 +------------------------+--------------------------------------------------+
91 | Parameters | Detail |
92 +========================+==================================================+
93 | -itd | -i: interactive, Keep STDIN open even if not |
95 | +--------------------------------------------------+
96 | | -t: allocate a pseudo-TTY detached mode, in the |
98 +------------------------+--------------------------------------------------+
99 | --privileged | If you want to build ``yardstick-image`` in |
100 | | Yardstick container, this parameter is needed |
101 +------------------------+--------------------------------------------------+
102 | -p 8888:5000 | Redirect the a host port (8888) to a container |
104 +------------------------+--------------------------------------------------+
105 | -v /var/run/docker.sock| If you want to use yardstick env |
106 | :/var/run/docker.sock | grafana/influxdb to create a grafana/influxdb |
107 | | container out of Yardstick container |
108 +------------------------+--------------------------------------------------+
109 | --name yardstick | The name for this container |
110 +------------------------+--------------------------------------------------+
113 If the host is restarted
114 ^^^^^^^^^^^^^^^^^^^^^^^^
116 The yardstick container must be started if the host is rebooted::
118 docker start yardstick
120 Configure the Yardstick container environment
121 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
123 There are three ways to configure environments for running Yardstick, explained
124 in the following sections. Before that, access the Yardstick container::
126 docker exec -it yardstick /bin/bash
128 and then configure Yardstick environments in the Yardstick container.
130 Using the CLI command ``env prepare`` (first way) (**recommended**)
131 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
133 In the Yardstick container, the Yardstick repository is located in the
134 ``/home/opnfv/repos`` directory. Yardstick provides a CLI to prepare OpenStack
135 environment variables and create Yardstick flavor and guest images
138 yardstick env prepare
140 .. note:: Since Euphrates release, the above command will not be able to
141 automatically configure the ``/etc/yardstick/openstack.creds`` file. So before
142 running the above command, it is necessary to create the
143 ``/etc/yardstick/openstack.creds`` file and save OpenStack environment
144 variables into it manually. If you have the openstack credential file saved
145 outside the Yardstick Docker container, you can do this easily by mapping the
146 credential file into Yardstick container using::
148 '-v /path/to/credential_file:/etc/yardstick/openstack.creds'
150 when running the Yardstick container. For details of the required OpenStack
151 environment variables please refer to section `Export OpenStack environment
154 The ``env prepare`` command may take up to 6-8 minutes to finish building
155 yardstick-image and other environment preparation. Meanwhile if you wish to
156 monitor the env prepare process, you can enter the Yardstick container in a new
157 terminal window and execute the following command::
159 tail -f /var/log/yardstick/uwsgi.log
162 Manually exporting the env variables and initializing OpenStack (second way)
163 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
165 Export OpenStack environment variables
166 ######################################
168 Before running Yardstick it is necessary to export OpenStack environment
173 Environment variables in the ``openrc`` file have to include at least::
181 A sample ``openrc`` file may look like this::
183 export OS_PASSWORD=console
184 export OS_PROJECT_NAME=admin
185 export OS_AUTH_URL=http://172.16.1.222:35357/v2.0
186 export OS_USERNAME=admin
187 export OS_VOLUME_API_VERSION=2
188 export EXTERNAL_NETWORK=net04_ext
191 Manual creation of Yardstick flavor and guest images
192 ####################################################
194 Before executing Yardstick test cases, make sure that Yardstick flavor and
195 guest image are available in OpenStack. Detailed steps about creating the
196 Yardstick flavor and building the Yardstick guest image can be found below.
198 Most of the sample test cases in Yardstick are using an OpenStack flavor called
199 ``yardstick-flavor`` which deviates from the OpenStack standard ``m1.tiny``
200 flavor by the disk size; instead of 1GB it has 3GB. Other parameters are the
201 same as in ``m1.tiny``.
203 Create ``yardstick-flavor``::
205 openstack flavor create --disk 3 --vcpus 1 --ram 512 --swap 100 \
208 Most of the sample test cases in Yardstick are using a guest image called
209 ``yardstick-image`` which deviates from an Ubuntu Cloud Server image
210 containing all the required tools to run test cases supported by Yardstick.
211 Yardstick has a tool for building this custom image. It is necessary to have
212 ``sudo`` rights to use this tool.
214 Also you may need install several additional packages to use this tool, by
215 follwing the commands below::
217 sudo -EH apt-get update && sudo -EH apt-get install -y qemu-utils kpartx
219 This image can be built using the following command in the directory where
220 Yardstick is installed::
222 export YARD_IMG_ARCH='amd64'
223 echo "Defaults env_keep += \'YARD_IMG_ARCH\'" | sudo tee --append \
224 /etc/sudoers > /dev/null
225 sudo -EH tools/yardstick-img-modify tools/ubuntu-server-cloudimg-modify.sh
227 .. warning:: Before building the guest image inside the Yardstick container,
228 make sure the container is granted with privilege. The script will create files
229 by default in ``/tmp/workspace/yardstick`` and the files will be owned by root.
231 The created image can be added to OpenStack using the OpenStack client or via
232 the OpenStack Dashboard::
234 openstack image create --disk-format qcow2 --container-format bare \
235 --public --file /tmp/workspace/yardstick/yardstick-image.img \
239 Some Yardstick test cases use a `Cirros 0.3.5`_ image and/or a `Ubuntu 16.04`_
240 image. Add Cirros and Ubuntu images to OpenStack::
242 openstack image create --disk-format qcow2 --container-format bare \
243 --public --file $cirros_image_file cirros-0.3.5
244 openstack image create --disk-format qcow2 --container-format bare \
245 --file $ubuntu_image_file Ubuntu-16.04
248 Automatic initialization of OpenStack (third way)
249 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
251 Similar to the second way, the first step is also to
252 `Export OpenStack environment variables`_. Then the following steps should be
255 Automatic creation of Yardstick flavor and guest images
256 #######################################################
258 Yardstick has a script for automatically creating Yardstick flavor and building
259 Yardstick guest images. This script is mainly used for CI and can be also used
260 in the local environment::
262 source $YARDSTICK_REPO_DIR/tests/ci/load_images.sh
265 The Yardstick container GUI
266 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
268 In Euphrates release, Yardstick implemented a GUI for Yardstick Docker
269 container. After booting up Yardstick container, you can visit the GUI at
270 ``<container_host_ip>:8888/gui/index.html``.
272 For usage of Yardstick GUI, please watch our demo video at
273 `Yardstick GUI demo`_.
275 .. note:: The Yardstick GUI is still in development, the GUI layout and
278 Delete the Yardstick container
279 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
281 If you want to uninstall Yardstick, just delete the Yardstick container::
283 sudo docker stop yardstick && docker rm yardstick
287 Install Yardstick directly in Ubuntu (second option)
288 ----------------------------------------------------
290 .. _install-framework:
292 Alternatively you can install Yardstick framework directly in Ubuntu or in an
293 Ubuntu Docker image. No matter which way you choose to install Yardstick, the
294 following installation steps are identical.
296 If you choose to use the Ubuntu Docker image, you can pull the Ubuntu
297 Docker image from Docker hub::
299 sudo -EH docker pull ubuntu:16.04
305 Prerequisite preparation::
307 sudo -EH apt-get update && sudo -EH apt-get install -y \
308 git python-setuptools python-pip
309 sudo -EH easy_install -U setuptools==30.0.0
310 sudo -EH pip install appdirs==1.4.0
311 sudo -EH pip install virtualenv
313 Download the source code and install Yardstick from it::
315 git clone https://gerrit.opnfv.org/gerrit/yardstick
316 export YARDSTICK_REPO_DIR=~/yardstick
318 sudo -EH ./install.sh
320 If the host is ever restarted, nginx and uwsgi need to be restarted::
322 service nginx restart
323 uwsgi -i /etc/yardstick/yardstick.ini
325 Configure the Yardstick environment (**Todo**)
326 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
328 For installing Yardstick directly in Ubuntu, the ``yardstick env`` command is
329 not available. You need to prepare OpenStack environment variables and create
330 Yardstick flavor and guest images manually.
336 For uninstalling Yardstick, just delete the virtual environment::
338 rm -rf ~/yardstick_venv
341 Install Yardstick directly in OpenSUSE
342 --------------------------------------
344 .. _install-framework:
346 You can install Yardstick framework directly in OpenSUSE.
352 Prerequisite preparation::
354 sudo -EH zypper -n install -y gcc \
367 python-setuptools-git
369 Create a virtual environment::
371 virtualenv ~/yardstick_venv
372 export YARDSTICK_VENV=~/yardstick_venv
373 source ~/yardstick_venv/bin/activate
374 sudo -EH easy_install -U setuptools
376 Download the source code and install Yardstick from it::
378 git clone https://gerrit.opnfv.org/gerrit/yardstick
379 export YARDSTICK_REPO_DIR=~/yardstick
381 sudo -EH python setup.py install
382 sudo -EH pip install -r requirements.txt
384 Install missing python modules::
386 sudo -EH pip install pyyaml \
393 python.glanceclient \
394 python.neutronclient \
399 Configure the Yardstick environment
400 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
402 Source the OpenStack environment variables::
404 source DEVSTACK_DIRECTORY/openrc
406 Export the Openstack external network. The default installation of Devstack
407 names the external network public::
409 export EXTERNAL_NETWORK=public
410 export OS_USERNAME=demo
412 Change the API version used by Yardstick to v2.0 (the devstack openrc sets it
415 export OS_AUTH_URL=http://PUBLIC_IP_ADDRESS:5000/v2.0
421 For unistalling Yardstick, just delete the virtual environment::
423 rm -rf ~/yardstick_venv
426 Verify the installation
427 -----------------------
429 It is recommended to verify that Yardstick was installed successfully
430 by executing some simple commands and test samples. Before executing Yardstick
431 test cases make sure ``yardstick-flavor`` and ``yardstick-image`` can be found
432 in OpenStack and the ``openrc`` file is sourced. Below is an example invocation
433 of Yardstick ``help`` command and ``ping.py`` test sample::
436 yardstick task start samples/ping.yaml
438 .. note:: The above commands could be run in both the Yardstick container and
441 Each testing tool supported by Yardstick has a sample configuration file.
442 These configuration files can be found in the ``samples`` directory.
444 Default location for the output is ``/tmp/yardstick.out``.
447 Automatic installation of Yardstick using ansible
448 -------------------------------------------------
450 Automatic installation can be used as an alternative to the manual.
451 Yardstick can be installed on the bare metal and to the container. Yardstick
452 container can be either pulled or built.
454 Bare metal installation
455 ^^^^^^^^^^^^^^^^^^^^^^^
457 Use ansible script ``install.yaml`` to install Yardstick on Ubuntu server:
459 .. code-block:: console
461 ansible-playbook -i install-inventory.ini install.yaml \
462 -e YARDSTICK_DIR=<path to Yardstick folder>
464 .. note:: By default ``INSTALLATION_MODE`` is ``baremetal``.
466 .. note:: By default Ubuntu 16.04 is chosen (xenial). It can be changed to
467 Ubuntu 18.04 (bionic) by passing ``-e OS_RELEASE=bionic`` parameter.
469 .. note:: To install Yardstick in virtual environment pass parameter
470 ``-e VIRTUAL_ENVIRONMENT=True``.
472 To build Yardstick NSB image pass ``IMG_PROPERTY=nsb`` as input parameter:
474 .. code-block:: console
476 ansible-playbook -i install-inventory.ini install.yaml \
477 -e IMAGE_PROPERTY=nsb \
478 -e YARDSTICK_DIR=<path to Yardstick folder>
480 .. note:: In this ``INSTALLATION_MODE`` mode either Yardstick image or SampleVNF
481 images will be built. Image type is defined by parameter ``IMAGE_PROPERTY``.
482 By default Yardstick image will be built.
484 Container installation
485 ^^^^^^^^^^^^^^^^^^^^^^
487 Use ansible script ``install.yaml`` to pull or build Yardstick
488 container. To pull Yardstick image and start container run:
490 .. code-block:: console
492 ansible-playbook -i install-inventory.ini install.yaml \
493 -e YARDSTICK_DIR=<path to Yardstick folder> \
494 -e INSTALLATION_MODE=container_pull
496 .. note:: In this ``INSTALLATION_MODE`` mode either Yardstick image or SampleVNF
497 images will be built. Image type is defined by variable ``IMG_PROPERTY`` in
498 file ``ansible/group_vars/all.yml``. By default Yardstick image will be
501 .. note:: Open question: How to know if Docker image is built on Ubuntu 16.04 and 18.04?
502 Do we need separate tag to be used?
504 To build Yardstick image run:
506 .. code-block:: console
508 ansible-playbook -i install-inventory.ini install.yaml \
509 -e YARDSTICK_DIR=<path to Yardstick folder> \
510 -e INSTALLATION_MODE=container
512 .. note:: In this ``INSTALLATION_MODE`` mode neither Yardstick image nor SampleVNF
515 .. note:: By default Ubuntu 16.04 is chosen (xenial). It can be changed to
516 Ubuntu 18.04 (bionic) by passing ``-e OS_RELEASE=bionic`` parameter.
518 Parameters for ``install.yaml``
519 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
521 Description of the parameters used with ``install.yaml`` script
523 +-------------------------+-------------------------------------------------+
524 | Parameters | Detail |
525 +=========================+=================================================+
526 | -i install-inventory.ini| Installs package dependency to remote servers |
527 | | Mandatory parameter |
528 | | By default no remote servers are provided |
529 | | Needed packages will be installed on localhost |
530 +-------------------------+-------------------------------------------------+
531 | -e YARDSTICK_DIR | Path to Yardstick folder |
532 | | Mandatory parameter |
533 +-------------------------+-------------------------------------------------+
534 | -e INSTALLATION_MODE | baremetal: Yardstick is installed to the bare |
536 | | Default parameter |
537 | +-------------------------------------------------+
538 | | container: Yardstick is installed in container |
539 | | Container is built from Dockerfile |
540 | +-------------------------------------------------+
541 | | container_pull: Yardstick is installed in |
543 | | Container is pulled from docker hub |
544 +-------------------------+-------------------------------------------------+
545 | -e OS_RELEASE | xenial or bionic: Ubuntu version to be used |
546 | | Default is Ubuntu 16.04 (xenial) |
547 +-------------------------+-------------------------------------------------+
548 | -e IMAGE_PROPERTY | normal or nsb: Type of the VM image to be built |
549 | | Default image is Yardstick |
550 +-------------------------+-------------------------------------------------+
551 | -e VIRTUAL_ENVIRONMENT | False or True: Whether install in virtualenv |
552 | | Default is False |
553 +-------------------------+-------------------------------------------------+
556 Deploy InfluxDB and Grafana using Docker
557 ----------------------------------------
559 Without InfluxDB, Yardstick stores results for running test case in the file
560 ``/tmp/yardstick.out``. However, it's inconvenient to retrieve and display
561 test results. So we will show how to use InfluxDB to store data and use
562 Grafana to display data in the following sections.
564 Automatic deployment of InfluxDB and Grafana containers (**recommended**)
565 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
567 1. Enter the Yardstick container::
569 sudo -EH docker exec -it yardstick /bin/bash
571 2. Create InfluxDB container and configure with the following command::
573 yardstick env influxdb
575 3. Create and configure Grafana container::
577 yardstick env grafana
579 Then you can run a test case and visit http://host_ip:1948
580 (``admin``/``admin``) to see the results.
582 .. note:: Executing ``yardstick env`` command to deploy InfluxDB and Grafana
583 requires Jumphost's docker API version => 1.24. Run the following command to
584 check the docker API version on the Jumphost::
589 Manual deployment of InfluxDB and Grafana containers
590 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
592 You can also deploy influxDB and Grafana containers manually on the Jumphost.
593 The following sections show how to do.
597 sudo -EH docker pull tutum/influxdb
598 sudo -EH docker pull grafana/grafana
602 sudo -EH docker run -d --name influxdb \
603 -p 8083:8083 -p 8086:8086 --expose 8090 --expose 8099 \
605 docker exec -it influxdb influx
609 > CREATE USER root WITH PASSWORD 'root' WITH ALL PRIVILEGES
610 > CREATE DATABASE yardstick;
617 sudo -EH docker run -d --name grafana -p 1948:3000 grafana/grafana
619 Log on to ``http://{YOUR_IP_HERE}:1948`` using ``admin``/``admin`` and configure
620 database resource to be ``{YOUR_IP_HERE}:8086``.
622 .. image:: images/Grafana_config.png
624 :alt: Grafana data source configuration
626 Configure ``yardstick.conf``::
628 sudo -EH docker exec -it yardstick /bin/bash
629 sudo cp etc/yardstick/yardstick.conf.sample /etc/yardstick/yardstick.conf
630 sudo vi /etc/yardstick/yardstick.conf
632 Modify ``yardstick.conf`` to add the ``influxdb`` dispatcher::
636 dispatcher = influxdb
638 [dispatcher_influxdb]
640 target = http://{YOUR_IP_HERE}:8086
645 Now Yardstick will store results in InfluxDB when you run a testcase.
648 Deploy InfluxDB and Grafana directly in Ubuntu (**Todo**)
649 ---------------------------------------------------------
655 To configure the Jumphost to access Internet through a proxy its necessary to
656 export several variables to the environment, contained in the following
660 _proxy=<proxy_address>
661 _proxyport=<proxy_port>
662 _ip=$(hostname -I | awk '{print $1}')
664 export ftp_proxy=http://$_proxy:$_proxyport
665 export FTP_PROXY=http://$_proxy:$_proxyport
666 export http_proxy=http://$_proxy:$_proxyport
667 export HTTP_PROXY=http://$_proxy:$_proxyport
668 export https_proxy=http://$_proxy:$_proxyport
669 export HTTPS_PROXY=http://$_proxy:$_proxyport
670 export no_proxy=127.0.0.1,localhost,$_ip,$(hostname),<.localdomain>
671 export NO_PROXY=127.0.0.1,localhost,$_ip,$(hostname),<.localdomain>
673 To enable Internet access from a container using ``docker``, depends on the OS
674 version. On Ubuntu 14.04 LTS, which uses SysVinit, ``/etc/default/docker`` must
678 # If you need Docker to use an HTTP proxy, it can also be specified here.
679 export http_proxy="http://<proxy_address>:<proxy_port>/"
680 export https_proxy="https://<proxy_address>:<proxy_port>/"
682 Then its necessary to restart the ``docker`` service::
684 sudo -EH service docker restart
686 In Ubuntu 16.04 LTS, which uses Systemd, its necessary to create a drop-in
689 sudo mkdir /etc/systemd/system/docker.service.d
691 Then, the proxy configuration will be stored in the following file::
693 # cat /etc/systemd/system/docker.service.d/http-proxy.conf
695 Environment="HTTP_PROXY=https://<proxy_address>:<proxy_port>/"
696 Environment="HTTPS_PROXY=https://<proxy_address>:<proxy_port>/"
697 Environment="NO_PROXY=localhost,127.0.0.1,<localaddress>,<.localdomain>"
699 The changes need to be flushed and the ``docker`` service restarted::
701 sudo systemctl daemon-reload
702 sudo systemctl restart docker
704 Any container is already created won't contain these modifications. If needed,
705 stop and delete the container::
707 sudo docker stop yardstick
708 sudo docker rm yardstick
710 .. warning:: Be careful, the above ``rm`` command will delete the container
711 completely. Everything on this container will be lost.
713 Then follow the previous instructions `Prepare the Yardstick container`_ to
714 rebuild the Yardstick container.
720 .. _`User Guide & Configuration Guide`: http://docs.opnfv.org/en/latest/release/userguide.introduction.html
721 .. _dockerhub: https://hub.docker.com/r/opnfv/yardstick/
722 .. _`Cirros 0.3.5`: http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img
723 .. _`Ubuntu 16.04`: https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img
724 .. _`Yardstick GUI demo`: https://www.youtube.com/watch?v=M3qbJDp6QBk