1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
3 .. (c) OPNFV, Intel Corporation and others.
5 ==========================
6 VES Application User Guide
7 ==========================
9 The Barometer repository contains a python based application for VES (VNF Event
10 Stream) which receives the `collectd`_ specific metrics via `Kafka`_ bus,
11 normalizes the metric data into the VES message and sends it into the VES
14 The application currently supports pushing platform relevant metrics through the
15 additional measurements field for VES.
17 Collectd has a ``write_kafka`` plugin that sends collectd metrics and values to
18 a Kafka Broker. The VES application uses Kafka Consumer to receive metrics
19 from the Kafka Broker.
25 1. Dependencies: install JAVA & Zookeeper.
31 $ sudo apt-get install default-jre
32 $ sudo apt-get install zookeeperd
33 $ sudo apt-get install python-pip
39 $ sudo yum install java-1.6.0-openjdk
40 $ sudo yum install python-pip
41 $ sudo yum install zookeeper
43 .. note:: You may need to add the repository that contains zookeeper.
44 To do so, follow the step below and try to install `zookeeper` again
45 using steps above. Otherwise, skip next step.
50 https://archive.cloudera.com/cdh5/one-click-install/redhat/7/x86_64/cloudera-cdh-5-0.x86_64.rpm
56 $ sudo zookeeper-server start
58 if you get the error message like ``ZooKeeper data directory is missing at /var/lib/zookeeper``
59 during the start of zookeeper, initialize zookeeper data directory using
60 the command below and start zookeeper again, otherwise skip the next step.
64 $ sudo /usr/lib/zookeeper/bin/zkServer-initialize.sh
65 No myid provided, be sure to specify it in /var/lib/zookeeper/myid if using non-standalone
67 2. Test if Zookeeper is running as a daemon.
71 $ telnet localhost 2181
73 Type 'ruok' & hit enter.
74 Expected response is 'imok' which means that Zookeeper is up running.
78 .. note:: VES doesn't work with version 0.9.4 of kafka-python. The
79 recommended/tested version is 1.3.3.
83 $ sudo pip install kafka-python
84 $ wget "https://archive.apache.org/dist/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz"
85 $ tar -xvzf kafka_2.11-0.11.0.0.tgz
86 $ sed -i -- 's/#delete.topic.enable=true/delete.topic.enable=true/' kafka_2.11-0.11.0.0/config/server.properties
87 $ sudo nohup kafka_2.11-0.11.0.0/bin/kafka-server-start.sh \
88 kafka_2.11-0.11.0.0/config/server.properties > kafka_2.11-0.11.0.0/kafka.log 2>&1 &
90 .. note:: If Kafka server fails to start, please check if the system IP
91 address is associated with the hostname in the static host lookup
92 table. If it doesn't exist, use the following command to add it.
96 $ echo "$(ip route get 8.8.8.8 | awk '{print $NF; exit}') $HOSTNAME" | sudo tee -a /etc/hosts
98 4. Test the Kafka Installation
100 To test if the installation worked correctly there are two scripts, producer and consumer scripts.
101 These will allow you to see messages pushed to broker and receive from broker.
103 Producer (Publish "Hello World"):
107 $ echo "Hello, World" | kafka_2.11-0.11.0.0/bin/kafka-console-producer.sh \
108 --broker-list localhost:9092 --topic TopicTest > /dev/null
110 Consumer (Receive "Hello World"):
114 $ kafka_2.11-0.11.0.0/bin/kafka-console-consumer.sh --zookeeper \
115 localhost:2181 --topic TopicTest --from-beginning --max-messages 1 --timeout-ms 3000
121 Install development tools:
127 $ sudo apt-get install build-essential bison autotools-dev autoconf
128 $ sudo apt-get install pkg-config flex libtool
134 $ sudo yum group install 'Development Tools'
136 .. The libkafka installed via the package manager may not work with collectd
137 (due to compilation issue). Thus, it's recommented to use the library installed
138 from sources using latest stable version of libkafka.
140 Install Apache Kafka C/C++ client library:
144 $ git clone https://github.com/edenhill/librdkafka.git ~/librdkafka
146 $ git checkout -b v0.9.5 v0.9.5
147 $ ./configure --prefix=/usr
151 Build collectd with Kafka support:
155 $ git clone https://github.com/collectd/collectd.git ~/collectd
158 $ ./configure --with-librdkafka=/usr --without-perl-bindings --enable-perl=no
159 $ make && sudo make install
161 Configure and start collectd. Create ``/opt/collectd/etc/collectd.conf``
162 collectd configuration file as following:
164 .. note:: The following collectd configuration file allows user to run VES
165 application in the guest mode. To run the VES in host mode, please follow
166 the `Configure VES in host mode`_ steps.
168 .. include:: collectd-ves-guest.conf
171 Start collectd process as a service as described in :ref:`install-collectd-as-a-service`.
174 Setup VES Test Collector
175 ------------------------
177 .. note:: Test Collector setup is required only for VES application testing
180 Install dependencies:
184 $ sudo pip install jsonschema
186 Clone VES Test Collector:
190 $ git clone https://github.com/att/evel-test-collector.git ~/evel-test-collector
192 Modify VES Test Collector config file to point to existing log directory and
197 $ sed -i.back 's/^\(log_file[ ]*=[ ]*\).*/\1collector.log/' ~/evel-test-collector/config/collector.conf
198 $ sed -i.back 's/^\(schema_file[ ]*=.*\)event_format_updated.json$/\1CommonEventFormat.json/' ~/evel-test-collector/config/collector.conf
200 Start VES Test Collector:
204 $ cd ~/evel-test-collector/code/collector
205 $ nohup python ./collector.py --config ../../config/collector.conf > collector.stdout.log &
208 Setup VES application (guest mode)
209 ----------------------------------
211 This mode is used to collect guest VM statistics provided by collectd
212 and send those metrics into the VES collector.
214 .. figure:: ves-app-guest-mode.png
218 Install dependencies:
222 $ sudo pip install pyyaml
224 Clone Barometer repo and start the VES application:
228 $ git clone https://gerrit.opnfv.org/gerrit/barometer ~/barometer
229 $ cd ~/barometer/3rd_party/collectd-ves-app/ves_app
230 $ nohup python ves_app.py --events-schema=guest.yaml --config=ves_app_config.conf > ves_app.stdout.log &
234 The above configuration is used for a localhost. The VES application can be
235 configured to use remote real VES collector and remote Kafka server. To do
236 so, the IP addresses/host names needs to be changed in ``collector.conf``
237 and ``ves_app_config.conf`` files accordingly.
240 Configure VES in host mode
241 --------------------------
243 This mode is used to collect hypervisor statistics about guest VMs and to send
244 those metrics into the VES collector. Also, this mode collects host statistics
245 and send them as part of the guest VES message.
247 .. figure:: ves-app-host-mode.png
251 Running the VES in host mode looks like steps described in
252 `Setup VES application (guest mode)`_ but with the following exceptions:
254 - The ``host.yaml`` configuration file should be used instead of ``guest.yaml``
255 file when VES application is running.
257 - Collectd should be running on host machine only.
259 - Addition ``libvirtd`` dependencies needs to be installed on a host where
260 collectd daemon is running. To install those dependencies, see :ref:`virt-plugin`
261 section of Barometer user guide.
263 - At least one VM instance should be up and running by hypervisor on the host.
265 - The next (minimum) configuration needs to be provided to collectd to be able
266 to generate the VES message to VES collector.
268 .. include:: collectd-ves-host.conf
271 to apply this configuration, the ``/opt/collectd/etc/collectd.conf`` file
272 needs to be modified based on example above and collectd daemon needs to
273 be restarted using the command below:
277 $ sudo systemctl restart collectd
279 .. note:: The list of the plugins can be extented depends on your needs.
282 VES application configuration description
283 -----------------------------------------
285 **Details of the Vendor Event Listener REST service**
287 REST resources are defined with respect to a ``ServerRoot``::
289 ServerRoot = https://{Domain}:{Port}/{optionalRoutingPath}
291 REST resources are of the form::
293 {ServerRoot}/eventListener/v{apiVersion}`
294 {ServerRoot}/eventListener/v{apiVersion}/{topicName}`
295 {ServerRoot}/eventListener/v{apiVersion}/eventBatch`
297 Within the VES directory (``3rd_party/collectd-ves-app/ves_app``) there is a
298 configuration file called ``ves_app.conf``. The description of the
299 configuration options are described below:
302 VES domain name. It can be IP address or hostname of VES collector
303 (default: ``127.0.0.1``)
306 VES port (default: ``30000``)
309 Used as the "optionalRoutingPath" element in the REST path (default: empty)
312 Used as the "topicName" element in the REST path (default: empty)
314 **UseHttps** *true|false*
315 Allow application to use HTTPS instead of HTTP (default: ``false``)
317 **Username** *"username"*
318 VES collector user name (default: empty)
320 **Password** *"passwd"*
321 VES collector password (default: empty)
323 **SendEventInterval** *interval*
324 This configuration option controls how often (sec) collectd data is sent to
325 Vendor Event Listener (default: ``20``)
327 **ApiVersion** *version*
328 Used as the "apiVersion" element in the REST path (default: ``5.1``)
331 Kafka Port (Default ``9092``)
333 **KafkaBroker** *host*
334 Kafka Broker domain name. It can be an IP address or hostname of local or remote server
335 (default: ``localhost``)
338 VES YAML configuration format
339 -----------------------------
341 The format of the VES message is generated by the VES application based on the
342 YAML schema configuration file provided by user via ``--events-schema``
343 command-line option of the application.
345 .. note:: Use ``--help`` option of VES application to see the
346 description of all available options
348 .. note:: The detailed installation guide of the VES application is described in
349 the `VES Application User Guide`_.
351 The message defined by YAML schema should correspond to format defined in
352 `VES shema definition`_.
354 .. warning:: This section doesn't explain the YAML language itself, so the
355 knowledge of the YAML language is required before continue reading it!
357 Since, the VES message output is a JSON format, it's recommended to understand
358 how YAML document is converted to JSON before starting to create a new YAML
359 definition for the VES. E.g.:
361 The following YAML document:
366 additionalMeasurements:
368 instance: someinstance
370 will be converted to JSON like this:
375 "additionalMeasurements": {
376 "instance": "someinstance",
381 .. note:: The `YAML syntax` section of `PyYAML documentation`_ can be used
382 as a reference for this.
388 The VES agent can generate two type of messages which are sent to the VES
389 collector. Each message type must be specified in the YAML configuration file
390 using a specific YAML tag.
393 This type is used to send a message defined in YAML configuration to the VES
394 collector with a specified interval (default is 20 sec and it's configurable
395 via command line option of the application). This type can be specified in
396 the configuration using ``!Measurements`` tag. For instance:
402 My Host Measurements: !Measurements
403 ... # message definition
406 This type is used to send a message defined in YAML configuration to the VES
407 collector when collectd notification is received from Kafka bus (collectd
408 ``write_kafka`` plugin). This type can be specified in the configuration
409 using ``!Events`` tag. For instance:
416 ... # event definition
419 Collectd metrics in VES
420 -----------------------
422 The VES application caches collectd metrics received via Kafka bus. The data
423 is stored in a table format. It's important to know it before mapping collectd
424 metric values to message defined in YAML configuration file.
426 VES collectd metric cache example:
428 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
429 | host | plugin | plugin_instance | type | type_instace | time | value | ds_name | interval |
430 +===========+===========+====================+===========+================+===================+========+=========+==========+
431 | localhost | cpu | | percent | user | 1509535512.30567 | 16 | value | 10 |
432 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
433 | localhost | memory | | memory | free | 1509535573.448014 | 798456 | value | 10 |
434 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
435 | localhost | interface | | eth0 | if_packets | 1509544183.956496 | 253 | rx | 10 |
436 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
437 | 7ec333e7 | virt | Ubuntu-12.04.5-LTS | percent | virt_cpu_total | 1509544402.378035 | 0.2 | value | 10 |
438 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
439 | 7ec333e7 | virt | Ubuntu-12.04.5-LTS | memory | rss | 1509544638.55119 | 123405 | value | 10 |
440 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
441 | 7ec333e7 | virt | Ubuntu-12.04.5-LTS | if_octets | vnet1 | 1509544646.27108 | 67 | tx | 10 |
442 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
443 | cc659a52 | virt | Ubuntu-16.04 | percent | virt_cpu_total | 1509544745.617207 | 0.3 | value | 10 |
444 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
445 | cc659a52 | virt | Ubuntu-16.04 | memory | rss | 1509544754.329411 | 4567 | value | 10 |
446 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
447 | cc659a52 | virt | Ubuntu-16.04 | if_octets | vnet0 | 1509544760.720381 | 0 | rx | 10 |
448 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
450 It's possible from YAML configuration file to refer to any field of any row of
451 the table via special YAML tags like ``ValueItem`` or ``ArrayItem``. See the
452 `Collectd metric reference`_ reference for more details.
454 .. note:: The `collectd data types file`_ contains map of ``type`` to
455 ``ds_name`` field. It can be used to get possible value for ``ds_name``
456 field. See the `collectd data types description`_ for more details on
460 Aging of collectd metric
461 ~~~~~~~~~~~~~~~~~~~~~~~~
463 If the metric will not be updated by the collectd during the double metric
464 interval time, it will be removed (aged) from VES internal cache.
470 There are four type of references supported by the YAML format: System,
471 Config, Collectd metric and Collectd notification. The format of the reference
476 "{<ref type>.<attribute name>}"
478 .. note:: Possible values for ``<ref type>`` and ``<attribute name>`` are
479 described in farther sections.
485 This reference is used to get system statistics like time, date etc. The system
486 reference (``<ref type>`` = ``system``) can be used in any place of the YAML
487 configuration file. This type of reference provides the following attributes:
490 The name of the host where VES application is running.
493 Unique ID during VES application runtime.
496 Current time in seconds since the Epoch. For example ``1509641411.631951``.
499 Current date in ISO 8601 format, ``YYYY-MM-DD``. For instance ``2017-11-02``.
506 Date: "{system.date}"
512 This reference is used to get VES configuration described in `VES application
513 configuration description`_ sectoin. The reference (``<ref type>`` = ``config``)
514 can be used in any place of the YAML configuration file. This type of reference
515 provides the following attributes:
518 Measurements dispatch interval. It referenses to ``SendEventInterval``
519 configuration of the VES application.
526 Interval: "{config.interval}"
529 Collectd metric reference
530 ~~~~~~~~~~~~~~~~~~~~~~~~~
532 This reference is used to get the attribute value of collectd metric from the
533 VES cache. The reference (``<ref type>`` = ``vl``) can be used ONLY inside
534 ``Measurements``, ``ValueItem`` and ``ArrayItem`` tags. Using the reference
535 inside a helper tag is also allowed if the helper tag is located inside the
536 tag where the reference is allowed (e.g.: ``ArrayItem``). The
537 ``<attribute name>`` name corresponds to the table field name described in
538 `Collectd metrics in VES`_ section. For example:
542 name: "{vl.type}-{vl.type_instance}"
545 Collectd notification reference
546 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
548 This reference is used to get the attribute value of received collectd
549 notification. The reference (``<ref type>`` = ``n``) can be used ONLY inside
550 ``Events`` tag. Using the reference inside a helper tag is also allowed if
551 the helper tag is located inside the ``Events`` tag. This type of reference
552 provides the following attributes:
555 The hostname of received collectd notification.
558 The plugin name of received collectd notification.
561 The plugin instance of the received collectd notification.
564 The type name of the received collectd notification.
567 The type instance of the received collectd notification.
570 The severity of the received collectd notification.
573 The message of the received collectd notification.
576 .. note:: The exact value for each attribute depends on the collectd plugin
577 which may generate the notification. Please refer to the
578 `collectd plugin description`_ document to get more details on the specific
585 sourceId: "{n.plugin_instance}"
588 Collectd metric mapping YAML tags
589 ---------------------------------
591 This section describes the YAML tags used to map collectd metric values in the
592 YAML configuration file.
598 This tag is a YAML map which is used to define the VES measurement message. It's
599 allowed to be used multiple times in the document (e.g.: you can define multiple
600 VES messages in one YAML document). This tag works in the same way as `ArrayItem
601 tag`_ does and all keys have the same description/rules.
607 This tag is used to select a collectd metric and get its attribute value using
608 `Collectd metric reference`_. The type of this tag is a YAML array of maps with
609 the possible keys described below.
611 ``SELECT`` (required)
612 Is a YAML map which describes the select metric criteria. Each key name of the
613 map must correspond to the table field name described in `Collectd metrics in VES`_
617 Describes the value to be assigned. If not provided, the default
618 ``!Number "{vl.value}"`` expression is used.
620 ``DEFAULT`` (optional)
621 Describes the default value which will be assigned if no metric is selected
622 by ``SELECT`` criteria.
625 ValueItem tag description example:
629 memoryFree: !ValueItem
634 - VALUE: !Bytes2Kibibytes "{vl.value}"
637 The tag process workflow is described on the figure below.
639 .. figure:: value-item-parse-workflow.png
641 YAML ValueItem tag process workflow
647 This tag is used to select a list of collectd metrics and generate a YAML array
648 of YAML items described by ``ITEM-DESC`` key. If no collectd metrics are
649 selected by the given criteria, the empty array will be returned.
651 ``SELECT`` (optional)
652 Is a YAML map which describes the select metrics criteria. Each key name of
653 the map must correspond to the table field name described in `Collectd
654 metrics in VES`_ section. The value of the key may be regular expression. To
655 enable regular expression in the value, the YAML string containing ``/`` char
656 at the beginning and at the end should be used. For example:
660 plugin: "/^(?!virt).*$/" # selected all metrics except ``virt`` plugin
662 The VES application uses the python RE library to work with regular
663 expression specified in the YAML configuration. Please refer to `python
664 regular expression syntax`_ documentation for more details on a syntax
667 Multiple ``SELECT`` keys are allowed by the tag. If nor ``SELECT`` or
668 ``INDEX-KEY`` key is specified, the VES error is generated.
670 ``INDEX-KEY`` (optional)
671 Is a YAML array which describes the unique fields to be selected by the tag.
672 Each element of array is a YAML string which should be one of the table field
673 name described in `Collectd metrics in VES`_ section. Please note, if this
674 key is used, only fields specified by the key can be used as a collectd
675 reference in the ``ITEM-DESC`` key.
677 ``ITEM-DESC`` (required)
678 Is a YAML map which describes each element of the YAML array generated by
679 the tag. Using ``ArrayItem`` tags and other `Helper YAML tags`_ are also
680 allowed in the definition of the key.
682 In the example below, the ArrayItem tag is used to generate an array of
683 ``ITEM-DESC`` items for each collectd metrics except ``virt`` plugin with
684 unique ``plugin``, ``plugin_instance`` attribute values.
688 Measurements: !ArrayItem
690 plugin: "/^(?!virt).*$/"
695 name: !StripExtraDash "{vl.plugin}-{vl.plugin_instance}"
697 The tag process workflow is described on the figure below.
699 .. figure:: array-item-parse-workflow.png
701 YAML ArrayItem tag process workflow
704 Collectd event mapping YAML tags
705 --------------------------------
707 This section describes the YAML tags used to map the collectd notification to
708 the VES event message in the YAML configuration file.
714 This tag is a YAML map which is used to define the VES event message. It's
715 allowed to be used multiple times in the document (e.g.: you can map multiple
716 collectd notification into VES message in one YAML document). The possible
717 keys of the tag are described below.
719 ``CONDITION`` (optional)
720 Is a YAML map which describes the select metric criteria. Each key name of
721 the map must correspond to the name of attribute provided by `Collectd
722 notification reference`_. If no such key provided, any collectd notification
723 will map the defined YAML message.
725 ``ITEM-DESC`` (required)
726 Is a YAML map which describes the message generated by this tag. Only `Helper
727 YAML tags`_ are allowed in the definition of the key.
729 The example of the VES event message which will be generate by the VES
730 application when collectd notification of the ``virt`` plugin is triggered
741 eventType: Notification
742 sourceId: &event_sourceId "{n.plugin_instance}"
743 sourceName: *event_sourceId
744 lastEpochMicrosec: !Number "{n.time}"
745 startEpochMicrosec: !Number "{n.time}"
747 alarmInterfaceA: !StripExtraDash "{n.plugin}-{n.plugin_instance}"
748 alarmCondition: "{n.severity}"
749 faultFieldsVersion: 1.1
757 This section describes the YAML tags used as utilities for formatting the output
758 message. The YAML configuration process workflow is described on the figure
761 .. figure:: parse-work-flow.png
763 YAML configuration process workflow
766 Convert string to number tag
767 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
769 The ``!Number`` tag is used in YAML configuration file to convert string value into
770 the number type. For instance:
774 lastEpochMicrosec: !Number "3456"
776 The output of the tag will be the JSON number.
781 lastEpochMicrosec: 3456
785 Convert bytes to Kibibytes tag
786 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
788 The ``!Bytes2Kibibytes`` tag is used in YAML configuration file to convert
789 bytes into kibibytes (1 kibibyte = 1024 bytes). For instance:
793 memoryConfigured: !Bytes2Kibibytes 4098
794 memoryConfigured: !Bytes2Kibibytes "1024"
796 The output of the tag will be the JSON number.
806 Convert one value to another tag
807 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
809 The ``!MapValue`` tag is used in YAML configuration file to map one value
810 into another value defined in the configuration. For instance:
820 The output of the tag will be the mapped value.
832 The ``!StripExtraDash`` tag is used in YAML configuration file to strip extra
833 dashes in the string (dashes at the beginning, at the end and double dashes).
838 name: !StripExtraDash string-with--extra-dashes-
840 The output of the tag will be the JSON string with extra dashes removed.
845 name: string-with-extra-dashes
852 #. Only one document can be defined in the same YAML configuration file.
854 #. The collectd notification is not supported by `Kafka collectd plugin`_. Due to
855 this limitation, the collectd notifications cannot be received by the VES
856 application and the defined YAML event will not be generated and sent to the
857 VES collector. Please note, that VES YAML format already supports the events
858 definition and the format is descibed in the document.
861 .. _collectd: http://collectd.org/
862 .. _Kafka: https://kafka.apache.org/
863 .. _`VES`: https://wiki.opnfv.org/display/fastpath/VES+plugin+updates
864 .. _`VES shema definition`: https://gerrit.onap.org/r/gitweb?p=demo.git;a=tree;f=vnfs/VES5.0/evel/evel-test-collector/docs/att_interface_definition;hb=refs/heads/master
865 .. _`PyYAML documentation`: https://pyyaml.org/wiki/PyYAMLDocumentation
866 .. _`collectd plugin description`: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod
867 .. _`collectd data types file`: https://github.com/collectd/collectd/blob/master/src/types.db
868 .. _`collectd data types description`: https://github.com/collectd/collectd/blob/master/src/types.db.pod
869 .. _`python regular expression syntax`: https://docs.python.org/2/library/re.html#regular-expression-syntax
870 .. _`Kafka collectd plugin`: https://collectd.org/wiki/index.php/Plugin:Write_Kafka