1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
3 .. (c) OPNFV, Intel Corporation and others.
5 ==========================
6 VES Application User Guide
7 ==========================
9 The Barometer repository contains a python based application for VES (VNF Event
10 Stream) which receives the `collectd`_ specific metrics via `Kafka`_ bus,
11 normalizes the metric data into the VES message and sends it into the VES
14 The application currently supports pushing platform relevant metrics through the
15 additional measurements field for VES.
17 Collectd has a ``write_kafka`` plugin that sends collectd metrics and values to
18 a Kafka Broker. The VES application uses Kafka Consumer to receive metrics
19 from the Kafka Broker.
25 1. Dependencies: install JAVA & Zookeeper.
31 $ sudo apt install default-jre
37 $ sudo yum install java-1.6.0-openjdk
43 $ sudo apt install zookeeperd
49 $ sudo yum install zookeeper
51 .. note:: You may need to add the repository that contains zookeeper.
52 To do so, follow the step below and try to install `zookeeper` again
53 using steps above. Otherwise, skip next step.
58 https://archive.cloudera.com/cdh5/one-click-install/redhat/7/x86_64/cloudera-cdh-5-0.x86_64.rpm
60 CentOS 7.x start zookeeper:
64 $ sudo zookeeper-server start
66 if you get the error message like ``ZooKeeper data directory is missing at /var/lib/zookeeper``
67 during the start of zookeeper, initialize zookeeper data directory using
68 the command below and start zookeeper again, otherwise skip the next step.
72 $ sudo /usr/lib/zookeeper/bin/zkServer-initialize.sh
73 No myid provided, be sure to specify it in /var/lib/zookeeper/myid if using non-standalone
75 To test if Zookeeper is running as a daemon.
79 $ telnet localhost 2181
81 Type 'ruok' & hit enter.
82 Expected response is 'imok'. Zookeeper is running fine.
86 VES doesn't work with version 0.9.4 of kafka-python.
87 The recommended/tested version is 1.3.3.
91 $ sudo yum install python-pip
92 $ sudo pip install kafka-python
98 $ wget "http://www-eu.apache.org/dist/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz"
100 3. Extract the archive:
104 $ tar -xvzf kafka_2.11-0.11.0.0.tgz
106 4. Configure Kafka Server:
110 $ vi kafka_2.11-0.11.0.0/config/server.properties
112 By default Kafka does not allow you to delete topics. Please uncomment:
116 delete.topic.enable=true
118 5. Start the Kafka Server.
120 Run 'kafka-server-start.sh' with nohup as a background process:
124 $ sudo nohup kafka_2.11-0.11.0.0/bin/kafka-server-start.sh \
125 kafka_2.11-0.11.0.0/config/server.properties > kafka_2.11-0.11.0.0/kafka.log 2>&1 &
127 6. Test Kafka Broker Installation
129 To test if the installation worked correctly there are two scripts, producer and consumer scripts.
130 These will allow you to see messages pushed to broker and receive from broker.
132 Producer (Publish "Hello World"):
136 $ echo "Hello, World" | kafka_2.11-0.11.0.0/bin/kafka-console-producer.sh \
137 --broker-list localhost:9092 --topic TopicTest > /dev/null
139 Consumer (Receive "Hello World"):
143 $ kafka_2.11-0.11.0.0/bin/kafka-console-consumer.sh --zookeeper \
144 localhost:2181 --topic TopicTest --from-beginning
150 Install development tools:
154 $ sudo yum group install 'Development Tools'
156 .. The libkafka installed via yum pkg manager is 0.11.0 which doesn't work with
157 collectd (compilation issue). Thus, we have to use the library installed
158 from sources using latest stable version which works with collectd.
160 Install Apache Kafka C/C++ client library:
164 $ git clone https://github.com/edenhill/librdkafka.git ~/librdkafka
166 $ git checkout -b v0.9.5 v0.9.5
167 $ ./configure --prefix=/usr
171 Build collectd with Kafka support:
175 $ git clone https://github.com/collectd/collectd.git ~/collectd
178 $ ./configure --with-librdkafka=/usr --without-perl-bindings --enable-perl=no
179 $ make && sudo make install
181 Configure and start collectd. Create ``/opt/collectd/etc/collectd.conf``
182 collectd configuration file as following:
184 .. note:: The following collectd configuration file allows user to run VES
185 application in the guest mode. To run the VES in host mode, please follow
186 the `Configure VES in host mode`_ steps.
188 .. include:: collectd-ves-guest.conf
191 Start collectd process as a service as described in :ref:`install-collectd-as-a-service`.
194 Setup VES Test Collector
195 ------------------------
197 .. note:: Test Collector setup is required only for VES application testing
200 Install dependencies:
204 $ sudo pip install jsonschema
206 Clone VES Test Collector:
210 $ git clone https://github.com/att/evel-test-collector.git ~/evel-test-collector
212 Modify VES Test Collector config file to point to existing log directory and
217 $ sed -i.back 's/^\(log_file[ ]*=[ ]*\).*/\1collector.log/' ~/evel-test-collector/config/collector.conf
218 $ sed -i.back 's/^\(schema_file[ ]*=.*\)event_format_updated.json$/\1CommonEventFormat.json/' ~/evel-test-collector/config/collector.conf
220 Start VES Test Collector:
224 $ cd ~/evel-test-collector/code/collector
225 $ nohup python ./collector.py --config ../../config/collector.conf > collector.stdout.log &
228 Setup VES application (guest mode)
229 ----------------------------------
231 Install dependencies:
235 $ sudo pip install pyyaml
237 Clone Barometer repo and start the VES application:
241 $ git clone https://gerrit.opnfv.org/gerrit/barometer ~/barometer
242 $ cd ~/barometer/3rd_party/collectd-ves-app/ves_app
243 $ nohup python ves_app.py --events-schema=guest.yaml --config=ves_app_config.conf > ves_app.stdout.log &
247 The above configuration is used for a localhost. The VES application can be
248 configured to use remote real VES collector and remote Kafka server. To do
249 so, the IP addresses/host names needs to be changed in ``collector.conf``
250 and ``ves_app_config.conf`` files accordingly.
253 Configure VES in host mode
254 --------------------------
256 Running the VES in host mode looks like steps described in
257 `Setup VES application (guest mode)`_ but with the following exceptions:
259 - The ``host.yaml`` configuration file should be used instead of ``guest.yaml``
260 file when VES application is running.
262 - Collectd should be running on host machine only.
264 - Addition ``libvirtd`` dependencies needs to be installed on a host where
265 collectd daemon is running. To install those dependencies, see :ref:`virt-plugin`
266 section of Barometer user guide.
268 - At least one VM instance should be up and running by hypervisor on the host.
270 - The next (minimum) configuration needs to be provided to collectd to be able
271 to generate the VES message to VES collector.
273 .. include:: collectd-ves-host.conf
276 to apply this configuration, the ``/opt/collectd/etc/collectd.conf`` file
277 needs to be modified based on example above and collectd daemon needs to
278 be restarted using the command below:
282 $ sudo systemctl restart collectd
284 .. note:: The list of the plugins can be extented depends on your needs.
287 VES application configuration description
288 -----------------------------------------
290 **Details of the Vendor Event Listener REST service**
292 REST resources are defined with respect to a ``ServerRoot``::
294 ServerRoot = https://{Domain}:{Port}/{optionalRoutingPath}
296 REST resources are of the form::
298 {ServerRoot}/eventListener/v{apiVersion}`
299 {ServerRoot}/eventListener/v{apiVersion}/{topicName}`
300 {ServerRoot}/eventListener/v{apiVersion}/eventBatch`
302 Within the VES directory (``3rd_party/collectd-ves-app/ves_app``) there is a
303 configuration file called ``ves_app.conf``. The description of the
304 configuration options are described below:
307 VES domain name. It can be IP address or hostname of VES collector
308 (default: ``127.0.0.1``)
311 VES port (default: ``30000``)
314 Used as the "optionalRoutingPath" element in the REST path (default: empty)
317 Used as the "topicName" element in the REST path (default: empty)
319 **UseHttps** *true|false*
320 Allow application to use HTTPS instead of HTTP (default: ``false``)
322 **Username** *"username"*
323 VES collector user name (default: empty)
325 **Password** *"passwd"*
326 VES collector password (default: empty)
328 **SendEventInterval** *interval*
329 This configuration option controls how often (sec) collectd data is sent to
330 Vendor Event Listener (default: ``20``)
332 **ApiVersion** *version*
333 Used as the "apiVersion" element in the REST path (default: ``5.1``)
336 Kafka Port (Default ``9092``)
338 **KafkaBroker** *host*
339 Kafka Broker domain name. It can be an IP address or hostname of local or remote server
340 (default: ``localhost``)
343 VES YAML configuration format
344 -----------------------------
346 The format of the VES message is generated by the VES application based on the
347 YAML schema configuration file provided by user via ``--events-schema``
348 command-line option of the application.
350 .. note:: Use ``--help`` option of VES application to see the
351 description of all available options
353 .. note:: The detailed installation guide of the VES application is described in
354 the `VES Application User Guide`_.
356 The message defined by YAML schema should correspond to format defined in
357 `VES shema definition`_.
359 .. warning:: This section doesn't explain the YAML language itself, so the
360 knowledge of the YAML language is required before continue reading it!
362 Since, the VES message output is a JSON format, it's recommended to understand
363 how YAML document is converted to JSON before starting to create a new YAML
364 definition for the VES. E.g.:
366 The following YAML document:
371 additionalMeasurements:
373 instance: someinstance
375 will be converted to JSON like this:
380 "additionalMeasurements": {
381 "instance": "someinstance",
386 .. note:: The `YAML syntax` section of `PyYAML documentation`_ can be used
387 as a reference for this.
393 The VES agent can generate two type of messages which are sent to the VES
394 collector. Each message type must be specified in the YAML configuration file
395 using a specific YAML tag.
398 This type is used to send a message defined in YAML configuration to the VES
399 collector with a specified interval (default is 20 sec and it's configurable
400 via command line option of the application). This type can be specified in
401 the configuration using ``!Measurements`` tag. For instance:
407 My Host Measurements: !Measurements
408 ... # message definition
411 This type is used to send a message defined in YAML configuration to the VES
412 collector when collectd notification is received from Kafka bus (collectd
413 ``write_kafka`` plugin). This type can be specified in the configuration
414 using ``!Events`` tag. For instance:
421 ... # event definition
424 Collectd metrics in VES
425 -----------------------
427 The VES application caches collectd metrics received via Kafka bus. The data
428 is stored in a table format. It's important to know it before mapping collectd
429 metric values to message defined in YAML configuration file.
431 VES collectd metric cache example:
433 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
434 | host | plugin | plugin_instance | type | type_instace | time | value | ds_name | interval |
435 +===========+===========+====================+===========+================+===================+========+=========+==========+
436 | localhost | cpu | | percent | user | 1509535512.30567 | 16 | value | 10 |
437 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
438 | localhost | memory | | memory | free | 1509535573.448014 | 798456 | value | 10 |
439 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
440 | localhost | interface | | eth0 | if_packets | 1509544183.956496 | 253 | rx | 10 |
441 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
442 | 7ec333e7 | virt | Ubuntu-12.04.5-LTS | percent | virt_cpu_total | 1509544402.378035 | 0.2 | value | 10 |
443 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
444 | 7ec333e7 | virt | Ubuntu-12.04.5-LTS | memory | rss | 1509544638.55119 | 123405 | value | 10 |
445 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
446 | 7ec333e7 | virt | Ubuntu-12.04.5-LTS | if_octets | vnet1 | 1509544646.27108 | 67 | tx | 10 |
447 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
448 | cc659a52 | virt | Ubuntu-16.04 | percent | virt_cpu_total | 1509544745.617207 | 0.3 | value | 10 |
449 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
450 | cc659a52 | virt | Ubuntu-16.04 | memory | rss | 1509544754.329411 | 4567 | value | 10 |
451 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
452 | cc659a52 | virt | Ubuntu-16.04 | if_octets | vnet0 | 1509544760.720381 | 0 | rx | 10 |
453 +-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
455 It's possible from YAML configuration file to refer to any field of any row of
456 the table via special YAML tags like ``ValueItem`` or ``ArrayItem``. See the
457 `Collectd metric reference`_ reference for more details.
459 .. note:: The `collectd data types file`_ contains map of ``type`` to
460 ``ds_name`` field. It can be used to get possible value for ``ds_name``
461 field. See the `collectd data types description`_ for more details on
465 Aging of collectd metric
466 ~~~~~~~~~~~~~~~~~~~~~~~~
468 If the metric will not be updated by the collectd during the double metric
469 interval time, it will be removed (aged) from VES internal cache.
475 There are four type of references supported by the YAML format: System,
476 Config, Collectd metric and Collectd notification. The format of the reference
481 "{<ref type>.<attribute name>}"
483 .. note:: Possible values for ``<ref type>`` and ``<attribute name>`` are
484 described in farther sections.
490 This reference is used to get system statistics like time, date etc. The system
491 reference (``<ref type>`` = ``system``) can be used in any place of the YAML
492 configuration file. This type of reference provides the following attributes:
495 The name of the host where VES application is running.
498 Unique ID during VES application runtime.
501 Current time in seconds since the Epoch. For example ``1509641411.631951``.
504 Current date in ISO 8601 format, ``YYYY-MM-DD``. For instance ``2017-11-02``.
511 Date: "{system.date}"
517 This reference is used to get VES configuration described in `VES application
518 configuration description`_ sectoin. The reference (``<ref type>`` = ``config``)
519 can be used in any place of the YAML configuration file. This type of reference
520 provides the following attributes:
523 Measurements dispatch interval. It referenses to ``SendEventInterval``
524 configuration of the VES application.
531 Interval: "{config.interval}"
534 Collectd metric reference
535 ~~~~~~~~~~~~~~~~~~~~~~~~~
537 This reference is used to get the attribute value of collectd metric from the
538 VES cache. The reference (``<ref type>`` = ``vl``) can be used ONLY inside
539 ``Measurements``, ``ValueItem`` and ``ArrayItem`` tags. Using the reference
540 inside a helper tag is also allowed if the helper tag is located inside the
541 tag where the reference is allowed (e.g.: ``ArrayItem``). The
542 ``<attribute name>`` name corresponds to the table field name described in
543 `Collectd metrics in VES`_ section. For example:
547 name: "{vl.type}-{vl.type_instance}"
550 Collectd notification reference
551 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
553 This reference is used to get the attribute value of received collectd
554 notification. The reference (``<ref type>`` = ``n``) can be used ONLY inside
555 ``Events`` tag. Using the reference inside a helper tag is also allowed if
556 the helper tag is located inside the ``Events`` tag. This type of reference
557 provides the following attributes:
560 The hostname of received collectd notification.
563 The plugin name of received collectd notification.
566 The plugin instance of the received collectd notification.
569 The type name of the received collectd notification.
572 The type instance of the received collectd notification.
575 The severity of the received collectd notification.
578 The message of the received collectd notification.
581 .. note:: The exact value for each attribute depends on the collectd plugin
582 which may generate the notification. Please refer to the
583 `collectd plugin description`_ document to get more details on the specific
590 sourceId: "{n.plugin_instance}"
593 Collectd metric mapping YAML tags
594 ---------------------------------
596 This section describes the YAML tags used to map collectd metric values in the
597 YAML configuration file.
603 This tag is a YAML map which is used to define the VES measurement message. It's
604 allowed to be used multiple times in the document (e.g.: you can define multiple
605 VES messages in one YAML document). This tag works in the same way as `ArrayItem
606 tag`_ does and all keys have the same description/rules.
612 This tag is used to select a collectd metric and get its attribute value using
613 `Collectd metric reference`_. The type of this tag is a YAML array of maps with
614 the possible keys described below.
616 ``SELECT`` (required)
617 Is a YAML map which describes the select metric criteria. Each key name of the
618 map must correspond to the table field name described in `Collectd metrics in VES`_
622 Describes the value to be assigned. If not provided, the default
623 ``!Number "{vl.value}"`` expression is used.
625 ``DEFAULT`` (optional)
626 Describes the default value which will be assigned if no metric is selected
627 by ``SELECT`` criteria.
630 ValueItem tag description example:
634 memoryFree: !ValueItem
639 - VALUE: !Bytes2Kibibytes "{vl.value}"
642 The tag process workflow is described on the figure below.
644 .. figure:: value-item-parse-workflow.png
646 YAML ValueItem tag process workflow
652 This tag is used to select a list of collectd metrics and generate a YAML array
653 of YAML items described by ``ITEM-DESC`` key. If no collectd metrics are
654 selected by the given criteria, the empty array will be returned.
656 ``SELECT`` (optional)
657 Is a YAML map which describes the select metrics criteria. Each key name of
658 the map must correspond to the table field name described in `Collectd
659 metrics in VES`_ section. The value of the key may be regular expression. To
660 enable regular expression in the value, the YAML string containing ``/`` char
661 at the beginning and at the end should be used. For example:
665 plugin: "/^(?!virt).*$/" # selected all metrics except ``virt`` plugin
667 The VES application uses the python RE library to work with regular
668 expression specified in the YAML configuration. Please refer to `python
669 regular expression syntax`_ documentation for more details on a syntax
672 Multiple ``SELECT`` keys are allowed by the tag. If nor ``SELECT`` or
673 ``INDEX-KEY`` key is specified, the VES error is generated.
675 ``INDEX-KEY`` (optional)
676 Is a YAML array which describes the unique fields to be selected by the tag.
677 Each element of array is a YAML string which should be one of the table field
678 name described in `Collectd metrics in VES`_ section. Please note, if this
679 key is used, only fields specified by the key can be used as a collectd
680 reference in the ``ITEM-DESC`` key.
682 ``ITEM-DESC`` (required)
683 Is a YAML map which describes each element of the YAML array generated by
684 the tag. Using ``ArrayItem`` tags and other `Helper YAML tags`_ are also
685 allowed in the definition of the key.
687 In the example below, the ArrayItem tag is used to generate an array of
688 ``ITEM-DESC`` items for each collectd metrics except ``virt`` plugin with
689 unique ``plugin``, ``plugin_instance`` attribute values.
693 Measurements: !ArrayItem
695 plugin: "/^(?!virt).*$/"
700 name: !StripExtraDash "{vl.plugin}-{vl.plugin_instance}"
702 The tag process workflow is described on the figure below.
704 .. figure:: array-item-parse-workflow.png
706 YAML ArrayItem tag process workflow
709 Collectd event mapping YAML tags
710 --------------------------------
712 This section describes the YAML tags used to map the collectd notification to
713 the VES event message in the YAML configuration file.
719 This tag is a YAML map which is used to define the VES event message. It's
720 allowed to be used multiple times in the document (e.g.: you can map multiple
721 collectd notification into VES message in one YAML document). The possible
722 keys of the tag are described below.
724 ``CONDITION`` (optional)
725 Is a YAML map which describes the select metric criteria. Each key name of
726 the map must correspond to the name of attribute provided by `Collectd
727 notification reference`_. If no such key provided, any collectd notification
728 will map the defined YAML message.
730 ``ITEM-DESC`` (required)
731 Is a YAML map which describes the message generated by this tag. Only `Helper
732 YAML tags`_ are allowed in the definition of the key.
734 The example of the VES event message which will be generate by the VES
735 application when collectd notification of the ``virt`` plugin is triggered
746 eventType: Notification
747 sourceId: &event_sourceId "{n.plugin_instance}"
748 sourceName: *event_sourceId
749 lastEpochMicrosec: !Number "{n.time}"
750 startEpochMicrosec: !Number "{n.time}"
752 alarmInterfaceA: !StripExtraDash "{n.plugin}-{n.plugin_instance}"
753 alarmCondition: "{n.severity}"
754 faultFieldsVersion: 1.1
762 This section describes the YAML tags used as utilities for formatting the output
763 message. The YAML configuration process workflow is described on the figure
766 .. figure:: parse-work-flow.png
768 YAML configuration process workflow
771 Convert string to number tag
772 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
774 The ``!Number`` tag is used in YAML configuration file to convert string value into
775 the number type. For instance:
779 lastEpochMicrosec: !Number "3456"
781 The output of the tag will be the JSON number.
786 lastEpochMicrosec: 3456
790 Convert bytes to Kibibytes tag
791 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
793 The ``!Bytes2Kibibytes`` tag is used in YAML configuration file to convert
794 bytes into kibibytes (1 kibibyte = 1024 bytes). For instance:
798 memoryConfigured: !Bytes2Kibibytes 4098
799 memoryConfigured: !Bytes2Kibibytes "1024"
801 The output of the tag will be the JSON number.
811 Convert one value to another tag
812 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
814 The ``!MapValue`` tag is used in YAML configuration file to map one value
815 into another value defined in the configuration. For instance:
825 The output of the tag will be the mapped value.
837 The ``!StripExtraDash`` tag is used in YAML configuration file to strip extra
838 dashes in the string (dashes at the beginning, at the end and double dashes).
843 name: !StripExtraDash string-with--extra-dashes-
845 The output of the tag will be the JSON string with extra dashes removed.
850 name: string-with-extra-dashes
857 #. Only one document can be defined in the same YAML configuration file.
859 #. The collectd notification is not supported by `Kafka collectd plugin`_. Due to
860 this limitation, the collectd notifications cannot be received by the VES
861 application and the defined YAML event will not be generated and sent to the
862 VES collector. Please note, that VES YAML format already supports the events
863 definition and the format is descibed in the document.
866 .. _collectd: http://collectd.org/
867 .. _Kafka: https://kafka.apache.org/
868 .. _`VES`: https://wiki.opnfv.org/display/fastpath/VES+plugin+updates
869 .. _`VES shema definition`: https://gerrit.onap.org/r/gitweb?p=demo.git;a=tree;f=vnfs/VES5.0/evel/evel-test-collector/docs/att_interface_definition;hb=refs/heads/master
870 .. _`PyYAML documentation`: https://pyyaml.org/wiki/PyYAMLDocumentation
871 .. _`collectd plugin description`: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod
872 .. _`collectd data types file`: https://github.com/collectd/collectd/blob/master/src/types.db
873 .. _`collectd data types description`: https://github.com/collectd/collectd/blob/master/src/types.db.pod
874 .. _`python regular expression syntax`: https://docs.python.org/2/library/re.html#regular-expression-syntax
875 .. _`Kafka collectd plugin`: https://collectd.org/wiki/index.php/Plugin:Write_Kafka