.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
-.. (c) OPNFV, Intel Corporation and others.
+.. (c) Anuket, Intel Corporation and others.
+.. _barometer-ves-userguide:
==========================
VES Application User Guide
The Barometer repository contains a python based application for VES (VNF Event
Stream) which receives the `collectd`_ specific metrics via `Kafka`_ bus,
-normalizes the metric data into the VES message and sends it into the VES
+normalizes the metric data into the VES message format and sends it into the VES
collector.
The application currently supports pushing platform relevant metrics through the
additional measurements field for VES.
Collectd has a ``write_kafka`` plugin that sends collectd metrics and values to
-a Kafka Broker. The VES application uses Kafka Consumer to receive metrics
-from the Kafka Broker.
+a Kafka Broker. The VES message formatting application, ves_app.py, receives metrics from
+the Kafka broker, normalises the data to VES message format for forwarding to VES collector.
+The VES message formatting application will be simply referred to as the "VES application"
+within this userguide
+The VES application can be run in host mode (baremetal), hypervisor mode (on a host with a hypervisor
+and VMs running) or guest mode(within a VM). The main software blocks that are required to run the
+VES application demo are:
+
+ 1. Kafka
+ 2. Collectd
+ 3. VES Application
+ 4. VES Collector
Install Kafka Broker
--------------------
1. Dependencies: install JAVA & Zookeeper.
- JAVA:
-
- .. code:: bash
-
- $ sudo apt install default-jre
-
- CentOS 7.x use:
-
- .. code:: bash
-
- $ sudo yum install java-1.6.0-openjdk
-
- Zookeeper:
+ Ubuntu 16.04:
.. code:: bash
- $ sudo apt install zookeeperd
+ $ sudo apt-get install default-jre
+ $ sudo apt-get install zookeeperd
+ $ sudo apt-get install python-pip
- CentOS 7.x use:
+ CentOS:
.. code:: bash
+ $ sudo yum update -y
+ $ sudo yum install java-1.8.0-openjdk
+ $ sudo yum install epel-release
+ $ sudo yum install python-pip
$ sudo yum install zookeeper
+ $ sudo yum install telnet
+ $ sudo yum install wget
.. note:: You may need to add the repository that contains zookeeper.
To do so, follow the step below and try to install `zookeeper` again
$ sudo yum install
https://archive.cloudera.com/cdh5/one-click-install/redhat/7/x86_64/cloudera-cdh-5-0.x86_64.rpm
- CentOS 7.x start zookeeper:
+ Start zookeeper:
.. code:: bash
.. code:: bash
- $ sudo /usr/lib/zookeeper/bin/zkServer-initialize.sh
+ $ sudo /usr/lib/zookeeper/bin/zkServer-initialize.sh
No myid provided, be sure to specify it in /var/lib/zookeeper/myid if using non-standalone
- To test if Zookeeper is running as a daemon.
+2. Test if Zookeeper is running as a daemon.
.. code:: bash
$ telnet localhost 2181
Type 'ruok' & hit enter.
- Expected response is 'imok'. Zookeeper is running fine.
+ Expected response is 'imok' which means that Zookeeper is up running.
- .. note::
+3. Install Kafka
- VES doesn't work with version 0.9.4 of kafka-python.
- The recommended/tested version is 1.3.3.
+ .. note:: VES doesn't work with version 0.9.4 of kafka-python. The
+ recommended/tested version is 1.3.3.
.. code:: bash
- $ sudo yum install python-pip
$ sudo pip install kafka-python
+ $ wget "https://archive.apache.org/dist/kafka/1.0.0/kafka_2.11-1.0.0.tgz"
+ $ tar -xvzf kafka_2.11-1.0.0.tgz
+ $ sed -i -- 's/#delete.topic.enable=true/delete.topic.enable=true/' kafka_2.11-1.0.0/config/server.properties
+ $ sudo nohup kafka_2.11-1.0.0/bin/kafka-server-start.sh \
+ kafka_2.11-1.0.0/config/server.properties > kafka_2.11-1.0.0/kafka.log 2>&1 &
-2. Download Kafka:
+ .. note:: If Kafka server fails to start, please check if the platform IP
+ address is associated with the hostname in the static host lookup
+ table. If it doesn't exist, use the following command to add it.
.. code:: bash
- $ wget "http://www-eu.apache.org/dist/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz"
-
-3. Extract the archive:
-
- .. code:: bash
-
- $ tar -xvzf kafka_2.11-0.11.0.0.tgz
-
-4. Configure Kafka Server:
-
- .. code:: bash
+ $ echo "$(ip route get 8.8.8.8 | awk '{print $NF; exit}') $HOSTNAME" | sudo tee -a /etc/hosts
- $ vi kafka_2.11-0.11.0.0/config/server.properties
-
- By default Kafka does not allow you to delete topics. Please uncomment:
-
- .. code:: bash
-
- delete.topic.enable=true
-
-5. Start the Kafka Server.
-
- Run 'kafka-server-start.sh' with nohup as a background process:
-
- .. code:: bash
-
- $ sudo nohup kafka_2.11-0.11.0.0/bin/kafka-server-start.sh \
- kafka_2.11-0.11.0.0/config/server.properties > kafka_2.11-0.11.0.0/kafka.log 2>&1 &
-
-6. Test Kafka Broker Installation
+4. Test the Kafka Installation
To test if the installation worked correctly there are two scripts, producer and consumer scripts.
These will allow you to see messages pushed to broker and receive from broker.
.. code:: bash
- $ echo "Hello, World" | kafka_2.11-0.11.0.0/bin/kafka-console-producer.sh \
+ $ echo "Hello, World" | kafka_2.11-1.0.0/bin/kafka-console-producer.sh \
--broker-list localhost:9092 --topic TopicTest > /dev/null
Consumer (Receive "Hello World"):
.. code:: bash
- $ kafka_2.11-0.11.0.0/bin/kafka-console-consumer.sh --zookeeper \
- localhost:2181 --topic TopicTest --from-beginning
+ $ kafka_2.11-1.0.0/bin/kafka-console-consumer.sh --zookeeper \
+ localhost:2181 --topic TopicTest --from-beginning --max-messages 1 --timeout-ms 3000
Install collectd
Install development tools:
-.. code:: bash
+ Ubuntu 16.04:
+
+ .. code:: bash
+
+ $ sudo apt-get install build-essential bison autotools-dev autoconf
+ $ sudo apt-get install pkg-config flex libtool
+
+ CentOS:
+
+ .. code:: bash
- $ sudo yum group install 'Development Tools'
+ $ sudo yum group install 'Development Tools'
-.. The libkafka installed via yum pkg manager is 0.11.0 which doesn't work with
- collectd (compilation issue). Thus, we have to use the library installed
- from sources using latest stable version which works with collectd.
+.. The libkafka installed via the package manager may not work with collectd
+ (due to compilation issue). Thus, it's recommented to use the library installed
+ from sources using latest stable version of libkafka.
Install Apache Kafka C/C++ client library:
$ ./configure --with-librdkafka=/usr --without-perl-bindings --enable-perl=no
$ make && sudo make install
-Configure and start collectd. Create ``/opt/collectd/etc/collectd.conf``
-collectd configuration file as following:
+.. note::
-.. note:: The following collectd configuration file allows user to run VES
- application in the guest mode. To run the VES in host mode, please follow
- the `Configure VES in host mode`_ steps.
+ If installing from git repository ``collectd.conf`` configuration file will be located in
+ directory ``/opt/collectd/etc/``. If installing from via a package manager ``collectd.conf``
+ configuration file will be located in directory ``/etc/collectd/``
-.. include:: collectd-ves-guest.conf
- :code: bash
+Configure and start collectd. Modify Collectd configuration file ``collectd.conf``
+as following:
+
+- Within a VM: `Setup VES application (guest mode)`_
+- On Host with VMs: `Setup VES application (hypervisor mode)`_
+- No Virtualization: `Setup VES application (host mode)`_
Start collectd process as a service as described in :ref:`install-collectd-as-a-service`.
+Setup VES application (guest mode)
+----------------------------------
-Setup VES Test Collector
-------------------------
+In this mode Collectd runs from within a VM and sends metrics to the VES collector.
-.. note:: Test Collector setup is required only for VES application testing
- purposes.
+.. figure:: ves-app-guest-mode.png
+
+ VES guest mode setup
Install dependencies:
.. code:: bash
- $ sudo pip install jsonschema
+ $ sudo pip install pyyaml python-kafka
-Clone VES Test Collector:
+Clone Barometer repo and start the VES application:
.. code:: bash
- $ git clone https://github.com/att/evel-test-collector.git ~/evel-test-collector
+ $ git clone https://gerrit.opnfv.org/gerrit/barometer
+ $ cd barometer/3rd_party/collectd-ves-app/ves_app
+ $ nohup python ves_app.py --events-schema=yaml/guest.yaml --config=config/ves_app_config.conf > ves_app.stdout.log &
-Modify VES Test Collector config file to point to existing log directory and
-schema file:
-.. code:: bash
+Modify Collectd configuration file ``collectd.conf`` as following:
- $ sed -i.back 's/^\(log_file[ ]*=[ ]*\).*/\1collector.log/' ~/evel-test-collector/config/collector.conf
- $ sed -i.back 's/^\(schema_file[ ]*=.*\)event_format_updated.json$/\1CommonEventFormat.json/' ~/evel-test-collector/config/collector.conf
+.. include:: collectd-ves-guest.conf
+ :code: bash
-Start VES Test Collector:
-.. code:: bash
+Start collectd process as a service as described in :ref:`install-collectd-as-a-service`.
- $ cd ~/evel-test-collector/code/collector
- $ nohup python ./collector.py --config ../../config/collector.conf > collector.stdout.log &
+.. note::
+ The above configuration is used for a localhost. The VES application can be
+ configured to use remote VES collector and remote Kafka server. To do
+ so, the IP addresses/host names needs to be changed in ``collector.conf``
+ and ``ves_app_config.conf`` files accordingly.
-Setup VES application (guest mode)
+
+Setup VES application (hypervisor mode)
+----------------------------------------
+
+This mode is used to collect hypervisor statistics about guest VMs and to send
+those metrics into the VES collector. Also, this mode collects host statistics
+and send them as part of the guest VES message.
+
+.. figure:: ves-app-hypervisor-mode.png
+
+ VES hypervisor mode setup
+
+Running the VES in hypervisor mode looks like steps described in
+`Setup VES application (guest mode)`_ but with the following exceptions:
+
+- The ``hypervisor.yaml`` configuration file should be used instead of ``guest.yaml``
+ file when VES application is running.
+
+- Collectd should be running on hypervisor machine only.
+
+- Addition ``libvirtd`` dependencies needs to be installed on where
+ collectd daemon is running. To install those dependencies, see :ref:`virt-plugin`
+ section of Barometer user guide.
+
+- The next (minimum) configuration needs to be provided to collectd to be able
+ to generate the VES message to VES collector.
+
+.. note::
+ At least one VM instance should be up and running by hypervisor on the host.
+
+.. include:: collectd-ves-hypervisor.conf
+ :code: bash
+
+
+Start collectd process as a service as described in :ref:`install-collectd-as-a-service`.
+
+.. note::
+
+ The above configuration is used for a localhost. The VES application can be
+ configured to use remote VES collector and remote Kafka server. To do
+ so, the IP addresses/host names needs to be changed in ``collector.conf``
+ and ``ves_app_config.conf`` files accordingly.
+
+
+.. note:: The list of the plugins can be extented depends on your needs.
+
+
+Setup VES application (host mode)
----------------------------------
+This mode is used to collect platform wide metrics and to send those metrics into the VES collector.
+It is most suitable for running within a baremetal platform.
+
Install dependencies:
.. code:: bash
.. code:: bash
- $ git clone https://gerrit.opnfv.org/gerrit/barometer ~/barometer
- $ cd ~/barometer/3rd_party/collectd-ves-app/ves_app
- $ nohup python ves_app.py --events-schema=guest.yaml --config=ves_app_config.conf > ves_app.stdout.log &
+ $ git clone https://gerrit.opnfv.org/gerrit/barometer
+ $ cd barometer/3rd_party/collectd-ves-app/ves_app
+ $ nohup python ves_app.py --events-schema=yaml/host.yaml --config=config/ves_app_config.conf > ves_app.stdout.log &
+
+
+.. figure:: ves-app-host-mode.png
+
+ VES Native mode setup
+
+Modify collectd configuration file ``collectd.conf`` as following:
+
+.. include:: collectd-ves-host.conf
+ :code: bash
+
+Start collectd process as a service as described in :ref:`install-collectd-as-a-service`.
.. note::
The above configuration is used for a localhost. The VES application can be
- configured to use remote real VES collector and remote Kafka server. To do
+ configured to use remote VES collector and remote Kafka server. To do
so, the IP addresses/host names needs to be changed in ``collector.conf``
and ``ves_app_config.conf`` files accordingly.
-Configure VES in host mode
---------------------------
+.. note:: The list of the plugins can be extented depends on your needs.
-Running the VES in host mode looks like steps described in
-`Setup VES application (guest mode)`_ but with the following exceptions:
-- The ``host.yaml`` configuration file should be used instead of ``guest.yaml``
- file when VES application is running.
+.. _Setup VES Test Collector:
-- Collectd should be running on host machine only.
+Setup VES Test Collector
+------------------------
-- Addition ``libvirtd`` dependencies needs to be installed on a host where
- collectd daemon is running. To install those dependencies, see :ref:`virt-plugin`
- section of Barometer user guide.
+.. note:: Test Collector setup is required only for VES application testing
+ purposes.
-- At least one VM instance should be up and running by hypervisor on the host.
+Install dependencies:
-- The next (minimum) configuration needs to be provided to collectd to be able
- to generate the VES message to VES collector.
+.. code:: bash
+
+ $ sudo pip install jsonschema
- .. include:: collectd-ves-host.conf
- :code: bash
+Clone VES Test Collector:
- to apply this configuration, the ``/opt/collectd/etc/collectd.conf`` file
- needs to be modified based on example above and collectd daemon needs to
- be restarted using the command below:
+.. code:: bash
- .. code:: bash
+ $ git clone https://github.com/att/evel-test-collector.git ~/evel-test-collector
- $ sudo systemctl restart collectd
+Modify VES Test Collector config file to point to existing log directory and
+schema file:
-.. note:: The list of the plugins can be extented depends on your needs.
+.. code:: bash
+
+ $ sed -i.back 's/^\(log_file[ ]*=[ ]*\).*/\1collector.log/' ~/evel-test-collector/config/collector.conf
+ $ sed -i.back 's/^\(schema_file[ ]*=.*\)event_format_updated.json$/\1CommonEventFormat.json/' ~/evel-test-collector/config/collector.conf
+
+Start VES Test Collector:
+
+.. code:: bash
+
+ $ cd ~/evel-test-collector/code/collector
+ $ nohup python ./collector.py --config ../../config/collector.conf > collector.stdout.log &
VES application configuration description
{ServerRoot}/eventListener/v{apiVersion}/{topicName}`
{ServerRoot}/eventListener/v{apiVersion}/eventBatch`
-Within the VES directory (``3rd_party/collectd-ves-app/ves_app``) there is a
-configuration file called ``ves_app.conf``. The description of the
+Within the VES directory (``3rd_party/collectd-ves-app/ves_app/config``) there is a
+configuration file called ``ves_app_conf.conf``. The description of the
configuration options are described below:
**Domain** *"host"*
VES port (default: ``30000``)
**Path** *"path"*
- Used as the "optionalRoutingPath" element in the REST path (default: empty)
+ Used as the "optionalRoutingPath" element in the REST path (default: ``vendor_event_listener``)
**Topic** *"path"*
- Used as the "topicName" element in the REST path (default: empty)
+ Used as the "topicName" element in the REST path (default: ``example_vnf``)
**UseHttps** *true|false*
Allow application to use HTTPS instead of HTTP (default: ``false``)
Vendor Event Listener (default: ``20``)
**ApiVersion** *version*
- Used as the "apiVersion" element in the REST path (default: ``5.1``)
+ Used as the "apiVersion" element in the REST path (default: ``3``)
**KafkaPort** *port*
Kafka Port (Default ``9092``)
definition and the format is descibed in the document.
-.. _collectd: http://collectd.org/
+.. _collectd: https://collectd.org/
.. _Kafka: https://kafka.apache.org/
-.. _`VES`: https://wiki.opnfv.org/display/fastpath/VES+plugin+updates
+.. _`VES`: https://wiki.anuket.io/display/HOME/VES+plugin+updates
.. _`VES shema definition`: https://gerrit.onap.org/r/gitweb?p=demo.git;a=tree;f=vnfs/VES5.0/evel/evel-test-collector/docs/att_interface_definition;hb=refs/heads/master
.. _`PyYAML documentation`: https://pyyaml.org/wiki/PyYAMLDocumentation
-.. _`collectd plugin description`: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod
-.. _`collectd data types file`: https://github.com/collectd/collectd/blob/master/src/types.db
-.. _`collectd data types description`: https://github.com/collectd/collectd/blob/master/src/types.db.pod
+.. _`collectd plugin description`: https://github.com/collectd/collectd/blob/main/src/collectd.conf.pod
+.. _`collectd data types file`: https://github.com/collectd/collectd/blob/main/src/types.db
+.. _`collectd data types description`: https://github.com/collectd/collectd/blob/main/src/types.db.pod
.. _`python regular expression syntax`: https://docs.python.org/2/library/re.html#regular-expression-syntax
.. _`Kafka collectd plugin`: https://collectd.org/wiki/index.php/Plugin:Write_Kafka
Code Review / barometer.git / blobdiff