==========================
VES Application User Guide
==========================
-The Barometer repository contains a python based application for VES.
+
+The Barometer repository contains a python based application for VES (VNF Event
+Stream) which receives the `collectd`_ specific metrics via `Kafka`_ bus,
+normalizes the metric data into the VES message and sends it into the VES
+collector.
The application currently supports pushing platform relevant metrics through the
additional measurements field for VES.
Start collectd process as a service as described in :ref:`install-collectd-as-a-service`.
-.. Start collectd process as a service as described in `Barometer User Guide
- <http://artifacts.opnfv.org/barometer/docs/index.html#installing-collectd-as-a-service>`_.
-
Setup VES Test Collector
------------------------
$ sudo pip install pyyaml
-Clone Barometer repo:
+Clone Barometer repo and start the VES application:
.. code:: bash
(default: ``localhost``)
-VES notification support
-------------------------
+VES YAML configuration format
+-----------------------------
+
+The format of the VES message is generated by the VES application based on the
+YAML schema configuration file provided by user via ``--events-schema``
+command-line option of the application.
+
+.. note:: Use ``--help`` option of VES application to see the
+ description of all available options
+
+.. note:: The detailed installation guide of the VES application is described in
+ the `VES Application User Guide`_.
+
+The message defined by YAML schema should correspond to format defined in
+`VES shema definition`_.
+
+.. warning:: This section doesn't explain the YAML language itself, so the
+ knowledge of the YAML language is required before continue reading it!
+
+Since, the VES message output is a JSON format, it's recommended to understand
+how YAML document is converted to JSON before starting to create a new YAML
+definition for the VES. E.g.:
+
+The following YAML document:
+
+.. code:: yaml
+
+ ---
+ additionalMeasurements:
+ plugin: somename
+ instance: someinstance
+
+will be converted to JSON like this:
+
+.. code:: json
+
+ {
+ "additionalMeasurements": {
+ "instance": "someinstance",
+ "plugin": "somename"
+ }
+ }
+
+.. note:: The `YAML syntax` section of `PyYAML documentation`_ can be used
+ as a reference for this.
+
+
+VES message types
+-----------------
+
+The VES agent can generate two type of messages which are sent to the VES
+collector. Each message type must be specified in the YAML configuration file
+using a specific YAML tag.
+
+Measurements
+ This type is used to send a message defined in YAML configuration to the VES
+ collector with a specified interval (default is 20 sec and it's configurable
+ via command line option of the application). This type can be specified in
+ the configuration using ``!Measurements`` tag. For instance:
+
+ .. code:: yaml
+
+ ---
+ # My comments
+ My Host Measurements: !Measurements
+ ... # message definition
+
+Events
+ This type is used to send a message defined in YAML configuration to the VES
+ collector when collectd notification is received from Kafka bus (collectd
+ ``write_kafka`` plugin). This type can be specified in the configuration
+ using ``!Events`` tag. For instance:
+
+ .. code:: yaml
+
+ ---
+ # My comments
+ My Events: !Events
+ ... # event definition
+
+
+Collectd metrics in VES
+-----------------------
+
+The VES application caches collectd metrics received via Kafka bus. The data
+is stored in a table format. It's important to know it before mapping collectd
+metric values to message defined in YAML configuration file.
+
+VES collectd metric cache example:
+
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| host | plugin | plugin_instance | type | type_instace | time | value | ds_name | interval |
++===========+===========+====================+===========+================+===================+========+=========+==========+
+| localhost | cpu | | percent | user | 1509535512.30567 | 16 | value | 10 |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| localhost | memory | | memory | free | 1509535573.448014 | 798456 | value | 10 |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| localhost | interface | | eth0 | if_packets | 1509544183.956496 | 253 | rx | 10 |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| 7ec333e7 | virt | Ubuntu-12.04.5-LTS | percent | virt_cpu_total | 1509544402.378035 | 0.2 | value | 10 |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| 7ec333e7 | virt | Ubuntu-12.04.5-LTS | memory | rss | 1509544638.55119 | 123405 | value | 10 |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| 7ec333e7 | virt | Ubuntu-12.04.5-LTS | if_octets | vnet1 | 1509544646.27108 | 67 | tx | 10 |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| cc659a52 | virt | Ubuntu-16.04 | percent | virt_cpu_total | 1509544745.617207 | 0.3 | value | 10 |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| cc659a52 | virt | Ubuntu-16.04 | memory | rss | 1509544754.329411 | 4567 | value | 10 |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+| cc659a52 | virt | Ubuntu-16.04 | if_octets | vnet0 | 1509544760.720381 | 0 | rx | 10 |
++-----------+-----------+--------------------+-----------+----------------+-------------------+--------+---------+----------+
+
+It's possible from YAML configuration file to refer to any field of any row of
+the table via special YAML tags like ``ValueItem`` or ``ArrayItem``. See the
+`Collectd metric reference`_ reference for more details.
+
+.. note:: The `collectd data types file`_ contains map of ``type`` to
+ ``ds_name`` field. It can be used to get possible value for ``ds_name``
+ field. See the `collectd data types description`_ for more details on
+ collectd data types.
+
+
+Aging of collectd metric
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+If the metric will not be updated by the collectd during the double metric
+interval time, it will be removed (aged) from VES internal cache.
+
+
+VES YAML references
+-------------------
+
+There are four type of references supported by the YAML format: System,
+Config, Collectd metric and Collectd notification. The format of the reference
+is the following:
+
+.. code:: yaml
+
+ "{<ref type>.<attribute name>}"
+
+.. note:: Possible values for ``<ref type>`` and ``<attribute name>`` are
+ described in farther sections.
+
+
+System reference
+~~~~~~~~~~~~~~~~
+
+This reference is used to get system statistics like time, date etc. The system
+reference (``<ref type>`` = ``system``) can be used in any place of the YAML
+configuration file. This type of reference provides the following attributes:
+
+``hostname``
+ The name of the host where VES application is running.
+
+``id``
+ Unique ID during VES application runtime.
+
+``time``
+ Current time in seconds since the Epoch. For example ``1509641411.631951``.
+
+``date``
+ Current date in ISO 8601 format, ``YYYY-MM-DD``. For instance ``2017-11-02``.
+
+
+For example:
+
+.. code:: yaml
+
+ Date: "{system.date}"
+
+
+Config reference
+~~~~~~~~~~~~~~~~
+
+This reference is used to get VES configuration described in `VES application
+configuration description`_ sectoin. The reference (``<ref type>`` = ``config``)
+can be used in any place of the YAML configuration file. This type of reference
+provides the following attributes:
+
+``interval``
+ Measurements dispatch interval. It referenses to ``SendEventInterval``
+ configuration of the VES application.
+
+
+For example:
+
+.. code:: yaml
+
+ Interval: "{config.interval}"
+
+
+Collectd metric reference
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This reference is used to get the attribute value of collectd metric from the
+VES cache. The reference (``<ref type>`` = ``vl``) can be used ONLY inside
+``Measurements``, ``ValueItem`` and ``ArrayItem`` tags. Using the reference
+inside a helper tag is also allowed if the helper tag is located inside the
+tag where the reference is allowed (e.g.: ``ArrayItem``). The
+``<attribute name>`` name corresponds to the table field name described in
+`Collectd metrics in VES`_ section. For example:
+
+.. code:: yaml
+
+ name: "{vl.type}-{vl.type_instance}"
+
+
+Collectd notification reference
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This reference is used to get the attribute value of received collectd
+notification. The reference (``<ref type>`` = ``n``) can be used ONLY inside
+``Events`` tag. Using the reference inside a helper tag is also allowed if
+the helper tag is located inside the ``Events`` tag. This type of reference
+provides the following attributes:
+
+``host``
+ The hostname of received collectd notification.
+
+``plugin``
+ The plugin name of received collectd notification.
+
+``plugin_instance``
+ The plugin instance of the received collectd notification.
+
+``type``
+ The type name of the received collectd notification.
+
+``type_instance``
+ The type instance of the received collectd notification.
+
+``severity``
+ The severity of the received collectd notification.
+
+``message``
+ The message of the received collectd notification.
+
+
+.. note:: The exact value for each attribute depends on the collectd plugin
+ which may generate the notification. Please refer to the
+ `collectd plugin description`_ document to get more details on the specific
+ collectd plugin.
+
+YAML config example:
+
+.. code:: yaml
+
+ sourceId: "{n.plugin_instance}"
+
+
+Collectd metric mapping YAML tags
+---------------------------------
+
+This section describes the YAML tags used to map collectd metric values in the
+YAML configuration file.
+
+
+Measurements tag
+~~~~~~~~~~~~~~~~
+
+This tag is a YAML map which is used to define the VES measurement message. It's
+allowed to be used multiple times in the document (e.g.: you can define multiple
+VES messages in one YAML document). This tag works in the same way as `ArrayItem
+tag`_ does and all keys have the same description/rules.
+
+
+ValueItem tag
+~~~~~~~~~~~~~
+
+This tag is used to select a collectd metric and get its attribute value using
+`Collectd metric reference`_. The type of this tag is a YAML array of maps with
+the possible keys described below.
+
+``SELECT`` (required)
+ Is a YAML map which describes the select metric criteria. Each key name of the
+ map must correspond to the table field name described in `Collectd metrics in VES`_
+ section.
+
+``VALUE`` (optional)
+ Describes the value to be assigned. If not provided, the default
+ ``!Number "{vl.value}"`` expression is used.
+
+``DEFAULT`` (optional)
+ Describes the default value which will be assigned if no metric is selected
+ by ``SELECT`` criteria.
+
+
+ValueItem tag description example:
+
+.. code:: yaml
+
+ memoryFree: !ValueItem
+ - SELECT:
+ plugin: memory
+ type: memory
+ type_instance: rss
+ - VALUE: !Bytes2Kibibytes "{vl.value}"
+ - DEFAULT: 0
+
+The tag process workflow is described on the figure below.
+
+.. figure:: value-item-parse-workflow.png
+
+ YAML ValueItem tag process workflow
+
+
+ArrayItem tag
+~~~~~~~~~~~~~
+
+This tag is used to select a list of collectd metrics and generate a YAML array
+of YAML items described by ``ITEM-DESC`` key. If no collectd metrics are
+selected by the given criteria, the empty array will be returned.
+
+``SELECT`` (optional)
+ Is a YAML map which describes the select metrics criteria. Each key name of
+ the map must correspond to the table field name described in `Collectd
+ metrics in VES`_ section. The value of the key may be regular expression. To
+ enable regular expression in the value, the YAML string containing ``/`` char
+ at the beginning and at the end should be used. For example:
+
+ .. code:: yaml
+
+ plugin: "/^(?!virt).*$/" # selected all metrics except ``virt`` plugin
+
+ The VES application uses the python RE library to work with regular
+ expression specified in the YAML configuration. Please refer to `python
+ regular expression syntax`_ documentation for more details on a syntax
+ used by the VES.
+
+ Multiple ``SELECT`` keys are allowed by the tag. If nor ``SELECT`` or
+ ``INDEX-KEY`` key is specified, the VES error is generated.
+
+``INDEX-KEY`` (optional)
+ Is a YAML array which describes the unique fields to be selected by the tag.
+ Each element of array is a YAML string which should be one of the table field
+ name described in `Collectd metrics in VES`_ section. Please note, if this
+ key is used, only fields specified by the key can be used as a collectd
+ reference in the ``ITEM-DESC`` key.
+
+``ITEM-DESC`` (required)
+ Is a YAML map which describes each element of the YAML array generated by
+ the tag. Using ``ArrayItem`` tags and other `Helper YAML tags`_ are also
+ allowed in the definition of the key.
+
+In the example below, the ArrayItem tag is used to generate an array of
+``ITEM-DESC`` items for each collectd metrics except ``virt`` plugin with
+unique ``plugin``, ``plugin_instance`` attribute values.
+
+.. code:: yaml
+
+ Measurements: !ArrayItem
+ - SELECT:
+ plugin: "/^(?!virt).*$/"
+ - INDEX-KEY:
+ - plugin
+ - plugin_instance
+ - ITEM-DESC:
+ name: !StripExtraDash "{vl.plugin}-{vl.plugin_instance}"
+
+The tag process workflow is described on the figure below.
+
+.. figure:: array-item-parse-workflow.png
+
+ YAML ArrayItem tag process workflow
+
+
+Collectd event mapping YAML tags
+--------------------------------
+
+This section describes the YAML tags used to map the collectd notification to
+the VES event message in the YAML configuration file.
+
+
+Events tag
+~~~~~~~~~~
+
+This tag is a YAML map which is used to define the VES event message. It's
+allowed to be used multiple times in the document (e.g.: you can map multiple
+collectd notification into VES message in one YAML document). The possible
+keys of the tag are described below.
+
+``CONDITION`` (optional)
+ Is a YAML map which describes the select metric criteria. Each key name of
+ the map must correspond to the name of attribute provided by `Collectd
+ notification reference`_. If no such key provided, any collectd notification
+ will map the defined YAML message.
+
+``ITEM-DESC`` (required)
+ Is a YAML map which describes the message generated by this tag. Only `Helper
+ YAML tags`_ are allowed in the definition of the key.
+
+The example of the VES event message which will be generate by the VES
+application when collectd notification of the ``virt`` plugin is triggered
+is described below.
+
+.. code:: yaml
+
+ ---
+ Virt Event: !Events
+ - ITEM-DESC:
+ event:
+ commonEventHeader:
+ domain: fault
+ eventType: Notification
+ sourceId: &event_sourceId "{n.plugin_instance}"
+ sourceName: *event_sourceId
+ lastEpochMicrosec: !Number "{n.time}"
+ startEpochMicrosec: !Number "{n.time}"
+ faultFields:
+ alarmInterfaceA: !StripExtraDash "{n.plugin}-{n.plugin_instance}"
+ alarmCondition: "{n.severity}"
+ faultFieldsVersion: 1.1
+ - CONDITION:
+ plugin: virt
+
+
+Helper YAML tags
+----------------
+
+This section describes the YAML tags used as utilities for formatting the output
+message. The YAML configuration process workflow is described on the figure
+below.
+
+.. figure:: parse-work-flow.png
+
+ YAML configuration process workflow
+
+
+Convert string to number tag
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``!Number`` tag is used in YAML configuration file to convert string value into
+the number type. For instance:
+
+.. code:: yaml
+
+ lastEpochMicrosec: !Number "3456"
+
+The output of the tag will be the JSON number.
+
+.. code:: json
+
+ {
+ lastEpochMicrosec: 3456
+ }
+
+
+Convert bytes to Kibibytes tag
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``!Bytes2Kibibytes`` tag is used in YAML configuration file to convert
+bytes into kibibytes (1 kibibyte = 1024 bytes). For instance:
+
+.. code:: yaml
+
+ memoryConfigured: !Bytes2Kibibytes 4098
+ memoryConfigured: !Bytes2Kibibytes "1024"
+
+The output of the tag will be the JSON number.
+
+.. code:: json
+
+ {
+ memoryConfigured: 4
+ memoryConfigured: 1
+ }
+
+
+Convert one value to another tag
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``!MapValue`` tag is used in YAML configuration file to map one value
+into another value defined in the configuration. For instance:
+
+.. code:: yaml
+
+ Severity: !MapValue
+ VALUE: Failure
+ TO:
+ Failure: Critical
+ Error: Warnign
+
+The output of the tag will be the mapped value.
+
+.. code:: json
+
+ {
+ Severity: Critical
+ }
+
+
+Strip extra dash tag
+~~~~~~~~~~~~~~~~~~~~
+
+The ``!StripExtraDash`` tag is used in YAML configuration file to strip extra
+dashes in the string (dashes at the beginning, at the end and double dashes).
+For example:
+
+.. code:: yaml
+
+ name: !StripExtraDash string-with--extra-dashes-
+
+The output of the tag will be the JSON string with extra dashes removed.
+
+.. code:: json
+
+ {
+ name: string-with-extra-dashes
+ }
+
+
+Limitations
+-----------
+
+#. Only one document can be defined in the same YAML configuration file.
+
+#. The collectd notification is not supported by `Kafka collectd plugin`_. Due to
+ this limitation, the collectd notifications cannot be received by the VES
+ application and the defined YAML event will not be generated and sent to the
+ VES collector. Please note, that VES YAML format already supports the events
+ definition and the format is descibed in the document.
+
-The VES application already supports YAML notification definitions but due to
-the collectd Kafka plugin limitations, collectd notifications cannot be received
-by the VES application. Thus, the VES notification (defined by YAML) will not be
-generated and sent to VES collector.
+.. _collectd: http://collectd.org/
+.. _Kafka: https://kafka.apache.org/
+.. _`VES`: https://wiki.opnfv.org/display/fastpath/VES+plugin+updates
+.. _`VES shema definition`: https://gerrit.onap.org/r/gitweb?p=demo.git;a=tree;f=vnfs/VES5.0/evel/evel-test-collector/docs/att_interface_definition;hb=refs/heads/master
+.. _`PyYAML documentation`: https://pyyaml.org/wiki/PyYAMLDocumentation
+.. _`collectd plugin description`: https://github.com/collectd/collectd/blob/master/src/collectd.conf.pod
+.. _`collectd data types file`: https://github.com/collectd/collectd/blob/master/src/types.db
+.. _`collectd data types description`: https://github.com/collectd/collectd/blob/master/src/types.db.pod
+.. _`python regular expression syntax`: https://docs.python.org/2/library/re.html#regular-expression-syntax
+.. _`Kafka collectd plugin`: https://collectd.org/wiki/index.php/Plugin:Write_Kafka
Code Review / barometer.git / commitdiff
