1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
3 .. (c) OPNFV, Intel Corporation and others.
5 ==========================
6 VES Application User Guide
7 ==========================
9 The Barometer repository contains a python based application for VES (VNF Event
10 Stream) which receives the `collectd`_ specific metrics via `Kafka`_ bus,
11 normalizes the metric data into the VES message and sends it into the VES
14 The application currently supports pushing platform relevant metrics through the
15 additional measurements field for VES.
17 Collectd has a ``write_kafka`` plugin that sends collectd metrics and values to
18 a Kafka Broker. The VES application uses Kafka Consumer to receive metrics
19 from the Kafka Broker.
25 1. Dependencies: install JAVA & Zookeeper.
31 $ sudo apt-get install default-jre
32 $ sudo apt-get install zookeeperd
33 $ sudo apt-get install python-pip
40 $ sudo yum install java-1.8.0-openjdk
41 $ sudo yum install epel-release
42 $ sudo yum install python-pip
43 $ sudo yum install zookeeper
44 $ sudo yum install telnet
45 $ sudo yum install wget
47 .. note:: You may need to add the repository that contains zookeeper.
48 To do so, follow the step below and try to install `zookeeper` again
49 using steps above. Otherwise, skip next step.
54 https://archive.cloudera.com/cdh5/one-click-install/redhat/7/x86_64/cloudera-cdh-5-0.x86_64.rpm
60 $ sudo zookeeper-server start
62 if you get the error message like ``ZooKeeper data directory is missing at /var/lib/zookeeper``
63 during the start of zookeeper, initialize zookeeper data directory using
64 the command below and start zookeeper again, otherwise skip the next step.
68 $ sudo /usr/lib/zookeeper/bin/zkServer-initialize.sh
69 No myid provided, be sure to specify it in /var/lib/zookeeper/myid if using non-standalone
71 2. Test if Zookeeper is running as a daemon.
75 $ telnet localhost 2181
77 Type 'ruok' & hit enter.
78 Expected response is 'imok' which means that Zookeeper is up running.
82 .. note:: VES doesn't work with version 0.9.4 of kafka-python. The
83 recommended/tested version is 1.3.3.
87 $ sudo pip install kafka-python
88 $ wget "https://archive.apache.org/dist/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz"
89 $ tar -xvzf kafka_2.11-0.11.0.0.tgz
90 $ sed -i -- 's/#delete.topic.enable=true/delete.topic.enable=true/' kafka_2.11-0.11.0.0/config/server.properties
91 $ sudo nohup kafka_2.11-0.11.0.0/bin/kafka-server-start.sh \
92 kafka_2.11-0.11.0.0/config/server.properties > kafka_2.11-0.11.0.0/kafka.log 2>&1 &
94 .. note:: If Kafka server fails to start, please check if the system IP
95 address is associated with the hostname in the static host lookup
96 table. If it doesn't exist, use the following command to add it.
100 $ echo "$(ip route get 8.8.8.8 | awk '{print $NF; exit}') $HOSTNAME" | sudo tee -a /etc/hosts
102 4. Test the Kafka Installation
104 To test if the installation worked correctly there are two scripts, producer and consumer scripts.
105 These will allow you to see messages pushed to broker and receive from broker.
107 Producer (Publish "Hello World"):
111 $ echo "Hello, World" | kafka_2.11-0.11.0.0/bin/kafka-console-producer.sh \
112 --broker-list localhost:9092 --topic TopicTest > /dev/null
114 Consumer (Receive "Hello World"):
118 $ kafka_2.11-0.11.0.0/bin/kafka-console-consumer.sh --zookeeper \
119 localhost:2181 --topic TopicTest --from-beginning --max-messages 1 --timeout-ms 3000
125 Install development tools:
131 $ sudo apt-get install build-essential bison autotools-dev autoconf
132 $ sudo apt-get install pkg-config flex libtool
138 $ sudo yum group install 'Development Tools'
140 .. The libkafka installed via the package manager may not work with collectd
141 (due to compilation issue). Thus, it's recommented to use the library installed
142 from sources using latest stable version of libkafka.
144 Install Apache Kafka C/C++ client library:
148 $ git clone https://github.com/edenhill/librdkafka.git ~/librdkafka
150 $ git checkout -b v0.9.5 v0.9.5
151 $ ./configure --prefix=/usr
155 Build collectd with Kafka support:
159 $ git clone https://github.com/collectd/collectd.git ~/collectd
162 $ ./configure --with-librdkafka=/usr --without-perl-bindings --enable-perl=no
163 $ make && sudo make install
165 Configure and start collectd. Create ``/opt/collectd/etc/collectd.conf``
166 collectd configuration file as following:
168 .. note:: The following collectd configuration file allows user to run VES
169 application in the guest mode. To run the VES in host mode, please follow
170 the `Configure VES in host mode`_ steps.
172 .. include:: collectd-ves-guest.conf
175 Start collectd process as a service as described in :ref:`install-collectd-as-a-service`.
178 Setup VES Test Collector
179 ------------------------
181 .. note:: Test Collector setup is required only for VES application testing
184 Install dependencies:
188 $ sudo pip install jsonschema
190 Clone VES Test Collector:
194 $ git clone https://github.com/att/evel-test-collector.git ~/evel-test-collector
196 Modify VES Test Collector config file to point to existing log directory and
201 $ sed -i.back 's/^\(log_file[ ]*=[ ]*\).*/\1collector.log/' ~/evel-test-collector/config/collector.conf
202 $ sed -i.back 's/^\(schema_file[ ]*=.*\)event_format_updated.json$/\1CommonEventFormat.json/' ~/evel-test-collector/config/collector.conf
204 Start VES Test Collector:
208 $ cd ~/evel-test-collector/code/collector
209 $ nohup python ./collector.py --config ../../config/collector.conf > collector.stdout.log &
212 Setup VES application (guest mode)
213 ----------------------------------
215 This mode is used to collect guest VM statistics provided by collectd
216 and send those metrics into the VES collector.
218 .. figure:: ves-app-guest-mode.png
222 Install dependencies:
226 $ sudo pip install pyyaml
228 Clone Barometer repo and start the VES application:
232 $ git clone https://gerrit.opnfv.org/gerrit/barometer ~/barometer
233 $ cd ~/barometer/3rd_party/collectd-ves-app/ves_app
234 $ nohup python ves_app.py --events-schema=guest.yaml --config=ves_app_config.conf > ves_app.stdout.log &
238 The above configuration is used for a localhost. The VES application can be
239 configured to use remote real VES collector and remote Kafka server. To do
240 so, the IP addresses/host names needs to be changed in ``collector.conf``
241 and ``ves_app_config.conf`` files accordingly.
244 Configure VES in host mode
245 --------------------------
247 This mode is used to collect hypervisor statistics about guest VMs and to send
248 those metrics into the VES collector. Also, this mode collects host statistics
249 and send them as part of the guest VES message.
251 .. figure:: ves-app-host-mode.png
255 Running the VES in host mode looks like steps described in
256 `Setup VES application (guest mode)`_ but with the following exceptions:
258 - The ``host.yaml`` configuration file should be used instead of ``guest.yaml``
259 file when VES application is running.
261 - Collectd should be running on host machine only.
263 - Addition ``libvirtd`` dependencies needs to be installed on a host where
264 collectd daemon is running. To install those dependencies, see :ref:`virt-plugin`
265 section of Barometer user guide.
267 - At least one VM instance should be up and running by hypervisor on the host.
269 - The next (minimum) configuration needs to be provided to collectd to be able
270 to generate the VES message to VES collector.
272 .. include:: collectd-ves-host.conf
275 to apply this configuration, the ``/opt/collectd/etc/collectd.conf`` file
276 needs to be modified based on example above and collectd daemon needs to
277 be restarted using the command below:
281 $ sudo systemctl restart collectd
283 .. note:: The list of the plugins can be extented depends on your needs.
286 VES application configuration description
287 -----------------------------------------
289 **Details of the Vendor Event Listener REST service**
291 REST resources are defined with respect to a ``ServerRoot``::
293 ServerRoot = https://{Domain}:{Port}/{optionalRoutingPath}
295 REST resources are of the form::
297 {ServerRoot}/eventListener/v{apiVersion}`
298 {ServerRoot}/eventListener/v{apiVersion}/{topicName}`
299 {ServerRoot}/eventListener/v{apiVersion}/eventBatch`
301 Within the VES directory (``3rd_party/collectd-ves-app/ves_app``) there is a
302 configuration file called ``ves_app.conf``. The description of the
303 configuration options are described below:
306 VES domain name. It can be IP address or hostname of VES collector
307 (default: ``127.0.0.1``)
310 VES port (default: ``30000``)
313 Used as the "optionalRoutingPath" element in the REST path (default: empty)
316 Used as the "topicName" element in the REST path (default: empty)
318 **UseHttps** *true|false*
319 Allow application to use HTTPS instead of HTTP (default: ``false``)
321 **Username** *"username"*
322 VES collector user name (default: empty)
324 **Password** *"passwd"*
325 VES collector password (default: empty)
327 **SendEventInterval** *interval*
328 This configuration option controls how often (sec) collectd data is sent to
329 Vendor Event Listener (default: ``20``)
331 **ApiVersion** *version*
332 Used as the "apiVersion" element in the REST path (default: ``5.1``)
335 Kafka Port (Default ``9092``)
337 **KafkaBroker** *host*
338 Kafka Broker domain name. It can be an IP address or hostname of local or remote server
339 (default: ``localhost``)
342 VES YAML configuration format
343 -----------------------------
345 The format of the VES message is generated by the VES application based on the
346 YAML schema configuration file provided by user via ``--events-schema``
347 command-line option of the application.
349 .. note:: Use ``--help`` option of VES application to see the
350 description of all available options
352 .. note:: The detailed installation guide of the VES application is described in
353 the `VES Application User Guide`_.
355 The message defined by YAML schema should correspond to format defined in
356 `VES shema definition`_.
358 .. warning:: This section doesn't explain the YAML language itself, so the
359 knowledge of the YAML language is required before continue reading it!
361 Since, the VES message output is a JSON format, it's recommended to understand
362 how YAML document is converted to JSON before starting to create a new YAML
363 definition for the VES. E.g.:
365 The following YAML document:
370 additionalMeasurements:
372 instance: someinstance
374 will be converted to JSON like this:
379 "additionalMeasurements": {
380 "instance": "someinstance",
385 .. note:: The `YAML syntax` section of `PyYAML documentation`_ can be used
386 as a reference for this.
392 The VES agent can generate two type of messages which are sent to the VES
393 collector. Each message type must be specified in the YAML configuration file
394 using a specific YAML tag.
397 This type is used to send a message defined in YAML configuration to the VES
398 collector with a specified interval (default is 20 sec and it's configurable
399 via command line option of the application). This type can be specified in
400 the configuration using ``!Measurements`` tag. For instance:
406 My Host Measurements: !Measurements
407 ... # message definition
410 This type is used to send a message defined in YAML configuration to the VES
411 collector when collectd notification is received from Kafka bus (collectd
412 ``write_kafka`` plugin). This type can be specified in the configuration
413 using ``!Events`` tag. For instance:
420 ... # event definition
423 Collectd metrics in VES
424 -----------------------
426 The VES application caches collectd metrics received via Kafka bus. The data
427 is stored in a table format. It's important to know it before mapping collectd
428 metric values to message defined in YAML configuration file.
430 VES collectd metric cache example:
432 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
433 | host | plugin | plugin_instance | type | type_instace | time | value | ds_name | interval |
434 +===========+===========+====================+===========+================+===================+========+=========+==========+
435 | localhost | cpu | | percent | user | 1509535512.30567 | 16 | value | 10 |
436 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
437 | localhost | memory | | memory | free | 1509535573.448014 | 798456 | value | 10 |
438 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
439 | localhost | interface | | eth0 | if_packets | 1509544183.956496 | 253 | rx | 10 |
440 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
441 | 7ec333e7 | virt | Ubuntu-12.04.5-LTS | percent | virt_cpu_total | 1509544402.378035 | 0.2 | value | 10 |
442 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
443 | 7ec333e7 | virt | Ubuntu-12.04.5-LTS | memory | rss | 1509544638.55119 | 123405 | value | 10 |
444 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
445 | 7ec333e7 | virt | Ubuntu-12.04.5-LTS | if_octets | vnet1 | 1509544646.27108 | 67 | tx | 10 |
446 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
447 | cc659a52 | virt | Ubuntu-16.04 | percent | virt_cpu_total | 1509544745.617207 | 0.3 | value | 10 |
448 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
449 | cc659a52 | virt | Ubuntu-16.04 | memory | rss | 1509544754.329411 | 4567 | value | 10 |
450 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
451 | cc659a52 | virt | Ubuntu-16.04 | if_octets | vnet0 | 1509544760.720381 | 0 | rx | 10 |
452 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
454 It's possible from YAML configuration file to refer to any field of any row of
455 the table via special YAML tags like ``ValueItem`` or ``ArrayItem``. See the
456 `Collectd metric reference`_ reference for more details.
458 .. note:: The `collectd data types file`_ contains map of ``type`` to
459 ``ds_name`` field. It can be used to get possible value for ``ds_name``
460 field. See the `collectd data types description`_ for more details on
464 Aging of collectd metric
465 ~~~~~~~~~~~~~~~~~~~~~~~~
467 If the metric will not be updated by the collectd during the double metric
468 interval time, it will be removed (aged) from VES internal cache.
474 There are four type of references supported by the YAML format: System,
475 Config, Collectd metric and Collectd notification. The format of the reference
480 "{<ref type>.<attribute name>}"
482 .. note:: Possible values for ``<ref type>`` and ``<attribute name>`` are
483 described in farther sections.
489 This reference is used to get system statistics like time, date etc. The system
490 reference (``<ref type>`` = ``system``) can be used in any place of the YAML
491 configuration file. This type of reference provides the following attributes:
494 The name of the host where VES application is running.
497 Unique ID during VES application runtime.
500 Current time in seconds since the Epoch. For example ``1509641411.631951``.
503 Current date in ISO 8601 format, ``YYYY-MM-DD``. For instance ``2017-11-02``.
510 Date: "{system.date}"
516 This reference is used to get VES configuration described in `VES application
517 configuration description`_ sectoin. The reference (``<ref type>`` = ``config``)
518 can be used in any place of the YAML configuration file. This type of reference
519 provides the following attributes:
522 Measurements dispatch interval. It referenses to ``SendEventInterval``
523 configuration of the VES application.
530 Interval: "{config.interval}"
533 Collectd metric reference
534 ~~~~~~~~~~~~~~~~~~~~~~~~~
536 This reference is used to get the attribute value of collectd metric from the
537 VES cache. The reference (``<ref type>`` = ``vl``) can be used ONLY inside
538 ``Measurements``, ``ValueItem`` and ``ArrayItem`` tags. Using the reference
539 inside a helper tag is also allowed if the helper tag is located inside the
540 tag where the reference is allowed (e.g.: ``ArrayItem``). The
541 ``<attribute name>`` name corresponds to the table field name described in
542 `Collectd metrics in VES`_ section. For example:
546 name: "{vl.type}-{vl.type_instance}"
549 Collectd notification reference
550 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
552 This reference is used to get the attribute value of received collectd
553 notification. The reference (``<ref type>`` = ``n``) can be used ONLY inside
554 ``Events`` tag. Using the reference inside a helper tag is also allowed if
555 the helper tag is located inside the ``Events`` tag. This type of reference
556 provides the following attributes:
559 The hostname of received collectd notification.
562 The plugin name of received collectd notification.
565 The plugin instance of the received collectd notification.
568 The type name of the received collectd notification.
571 The type instance of the received collectd notification.
574 The severity of the received collectd notification.
577 The message of the received collectd notification.
580 .. note:: The exact value for each attribute depends on the collectd plugin
581 which may generate the notification. Please refer to the
582 `collectd plugin description`_ document to get more details on the specific
589 sourceId: "{n.plugin_instance}"
592 Collectd metric mapping YAML tags
593 ---------------------------------
595 This section describes the YAML tags used to map collectd metric values in the
596 YAML configuration file.
602 This tag is a YAML map which is used to define the VES measurement message. It's
603 allowed to be used multiple times in the document (e.g.: you can define multiple
604 VES messages in one YAML document). This tag works in the same way as `ArrayItem
605 tag`_ does and all keys have the same description/rules.
611 This tag is used to select a collectd metric and get its attribute value using
612 `Collectd metric reference`_. The type of this tag is a YAML array of maps with
613 the possible keys described below.
615 ``SELECT`` (required)
616 Is a YAML map which describes the select metric criteria. Each key name of the
617 map must correspond to the table field name described in `Collectd metrics in VES`_
621 Describes the value to be assigned. If not provided, the default
622 ``!Number "{vl.value}"`` expression is used.
624 ``DEFAULT`` (optional)
625 Describes the default value which will be assigned if no metric is selected
626 by ``SELECT`` criteria.
629 ValueItem tag description example:
633 memoryFree: !ValueItem
638 - VALUE: !Bytes2Kibibytes "{vl.value}"
641 The tag process workflow is described on the figure below.
643 .. figure:: value-item-parse-workflow.png
645 YAML ValueItem tag process workflow
651 This tag is used to select a list of collectd metrics and generate a YAML array
652 of YAML items described by ``ITEM-DESC`` key. If no collectd metrics are
653 selected by the given criteria, the empty array will be returned.
655 ``SELECT`` (optional)
656 Is a YAML map which describes the select metrics criteria. Each key name of
657 the map must correspond to the table field name described in `Collectd
658 metrics in VES`_ section. The value of the key may be regular expression. To
659 enable regular expression in the value, the YAML string containing ``/`` char
660 at the beginning and at the end should be used. For example:
664 plugin: "/^(?!virt).*$/" # selected all metrics except ``virt`` plugin
666 The VES application uses the python RE library to work with regular
667 expression specified in the YAML configuration. Please refer to `python
668 regular expression syntax`_ documentation for more details on a syntax
671 Multiple ``SELECT`` keys are allowed by the tag. If nor ``SELECT`` or
672 ``INDEX-KEY`` key is specified, the VES error is generated.
674 ``INDEX-KEY`` (optional)
675 Is a YAML array which describes the unique fields to be selected by the tag.
676 Each element of array is a YAML string which should be one of the table field
677 name described in `Collectd metrics in VES`_ section. Please note, if this
678 key is used, only fields specified by the key can be used as a collectd
679 reference in the ``ITEM-DESC`` key.
681 ``ITEM-DESC`` (required)
682 Is a YAML map which describes each element of the YAML array generated by
683 the tag. Using ``ArrayItem`` tags and other `Helper YAML tags`_ are also
684 allowed in the definition of the key.
686 In the example below, the ArrayItem tag is used to generate an array of
687 ``ITEM-DESC`` items for each collectd metrics except ``virt`` plugin with
688 unique ``plugin``, ``plugin_instance`` attribute values.
692 Measurements: !ArrayItem
694 plugin: "/^(?!virt).*$/"
699 name: !StripExtraDash "{vl.plugin}-{vl.plugin_instance}"
701 The tag process workflow is described on the figure below.
703 .. figure:: array-item-parse-workflow.png
705 YAML ArrayItem tag process workflow
708 Collectd event mapping YAML tags
709 --------------------------------
711 This section describes the YAML tags used to map the collectd notification to
712 the VES event message in the YAML configuration file.
718 This tag is a YAML map which is used to define the VES event message. It's
719 allowed to be used multiple times in the document (e.g.: you can map multiple
720 collectd notification into VES message in one YAML document). The possible
721 keys of the tag are described below.
723 ``CONDITION`` (optional)
724 Is a YAML map which describes the select metric criteria. Each key name of
725 the map must correspond to the name of attribute provided by `Collectd
726 notification reference`_. If no such key provided, any collectd notification
727 will map the defined YAML message.
729 ``ITEM-DESC`` (required)
730 Is a YAML map which describes the message generated by this tag. Only `Helper
731 YAML tags`_ are allowed in the definition of the key.
733 The example of the VES event message which will be generate by the VES
734 application when collectd notification of the ``virt`` plugin is triggered
745 eventType: Notification
746 sourceId: &event_sourceId "{n.plugin_instance}"
747 sourceName: *event_sourceId
748 lastEpochMicrosec: !Number "{n.time}"
749 startEpochMicrosec: !Number "{n.time}"
751 alarmInterfaceA: !StripExtraDash "{n.plugin}-{n.plugin_instance}"
752 alarmCondition: "{n.severity}"
753 faultFieldsVersion: 1.1
761 This section describes the YAML tags used as utilities for formatting the output
762 message. The YAML configuration process workflow is described on the figure
765 .. figure:: parse-work-flow.png
767 YAML configuration process workflow
770 Convert string to number tag
771 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
773 The ``!Number`` tag is used in YAML configuration file to convert string value into
774 the number type. For instance:
778 lastEpochMicrosec: !Number "3456"
780 The output of the tag will be the JSON number.
785 lastEpochMicrosec: 3456
789 Convert bytes to Kibibytes tag
790 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
792 The ``!Bytes2Kibibytes`` tag is used in YAML configuration file to convert
793 bytes into kibibytes (1 kibibyte = 1024 bytes). For instance:
797 memoryConfigured: !Bytes2Kibibytes 4098
798 memoryConfigured: !Bytes2Kibibytes "1024"
800 The output of the tag will be the JSON number.
810 Convert one value to another tag
811 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
813 The ``!MapValue`` tag is used in YAML configuration file to map one value
814 into another value defined in the configuration. For instance:
824 The output of the tag will be the mapped value.
836 The ``!StripExtraDash`` tag is used in YAML configuration file to strip extra
837 dashes in the string (dashes at the beginning, at the end and double dashes).
842 name: !StripExtraDash string-with--extra-dashes-
844 The output of the tag will be the JSON string with extra dashes removed.
849 name: string-with-extra-dashes
856 #. Only one document can be defined in the same YAML configuration file.
858 #. The collectd notification is not supported by `Kafka collectd plugin`_. Due to
859 this limitation, the collectd notifications cannot be received by the VES
860 application and the defined YAML event will not be generated and sent to the
861 VES collector. Please note, that VES YAML format already supports the events
862 definition and the format is descibed in the document.
865 .. _collectd: http://collectd.org/
866 .. _Kafka: https://kafka.apache.org/
867 .. _`VES`: https://wiki.opnfv.org/display/fastpath/VES+plugin+updates
868 .. _`VES shema definition`: https://gerrit.onap.org/r/gitweb?p=demo.git;a=tree;f=vnfs/VES5.0/evel/evel-test-collector/docs/att_interface_definition;hb=refs/heads/master
869 .. _`PyYAML documentation`: https://pyyaml.org/wiki/PyYAMLDocumentation
870 .. _`collectd plugin description`: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod
871 .. _`collectd data types file`: https://github.com/collectd/collectd/blob/master/src/types.db
872 .. _`collectd data types description`: https://github.com/collectd/collectd/blob/master/src/types.db.pod
873 .. _`python regular expression syntax`: https://docs.python.org/2/library/re.html#regular-expression-syntax
874 .. _`Kafka collectd plugin`: https://collectd.org/wiki/index.php/Plugin:Write_Kafka