1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
3 .. (c) OPNFV, Intel Corporation and others.
5 ==========================
6 VES Application User Guide
7 ==========================
8 The Barometer repository contains a python based application for VES.
10 The application currently supports pushing platform relevant metrics through the
11 additional measurements field for VES.
13 Collectd has a ``write_kafka`` plugin that sends collectd metrics and values to
14 a Kafka Broker. The VES application uses Kafka Consumer to receive metrics
15 from the Kafka Broker.
21 1. Dependencies: install JAVA & Zookeeper.
27 $ sudo apt install default-jre
33 $ sudo yum install java-1.6.0-openjdk
39 $ sudo apt install zookeeperd
45 $ sudo yum install zookeeper
47 .. note:: You may need to add the repository that contains zookeeper.
48 To do so, follow the step below and try to install `zookeeper` again
49 using steps above. Otherwise, skip next step.
54 https://archive.cloudera.com/cdh5/one-click-install/redhat/7/x86_64/cloudera-cdh-5-0.x86_64.rpm
56 CentOS 7.x start zookeeper:
60 $ sudo zookeeper-server start
62 if you get the error message like ``ZooKeeper data directory is missing at /var/lib/zookeeper``
63 during the start of zookeeper, initialize zookeeper data directory using
64 the command below and start zookeeper again, otherwise skip the next step.
68 $ sudo /usr/lib/zookeeper/bin/zkServer-initialize.sh
69 No myid provided, be sure to specify it in /var/lib/zookeeper/myid if using non-standalone
71 To test if Zookeeper is running as a daemon.
75 $ telnet localhost 2181
77 Type 'ruok' & hit enter.
78 Expected response is 'imok'. Zookeeper is running fine.
82 VES doesn't work with version 0.9.4 of kafka-python.
83 The recommended/tested version is 1.3.3.
87 $ sudo yum install python-pip
88 $ sudo pip install kafka-python
94 $ wget "http://www-eu.apache.org/dist/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz"
96 3. Extract the archive:
100 $ tar -xvzf kafka_2.11-0.11.0.0.tgz
102 4. Configure Kafka Server:
106 $ vi kafka_2.11-0.11.0.0/config/server.properties
108 By default Kafka does not allow you to delete topics. Please uncomment:
112 delete.topic.enable=true
114 5. Start the Kafka Server.
116 Run 'kafka-server-start.sh' with nohup as a background process:
120 $ sudo nohup kafka_2.11-0.11.0.0/bin/kafka-server-start.sh \
121 kafka_2.11-0.11.0.0/config/server.properties > kafka_2.11-0.11.0.0/kafka.log 2>&1 &
123 6. Test Kafka Broker Installation
125 To test if the installation worked correctly there are two scripts, producer and consumer scripts.
126 These will allow you to see messages pushed to broker and receive from broker.
128 Producer (Publish "Hello World"):
132 $ echo "Hello, World" | kafka_2.11-0.11.0.0/bin/kafka-console-producer.sh \
133 --broker-list localhost:9092 --topic TopicTest > /dev/null
135 Consumer (Receive "Hello World"):
139 $ kafka_2.11-0.11.0.0/bin/kafka-console-consumer.sh --zookeeper \
140 localhost:2181 --topic TopicTest --from-beginning
146 Install development tools:
150 $ sudo yum group install 'Development Tools'
152 .. The libkafka installed via yum pkg manager is 0.11.0 which doesn't work with
153 collectd (compilation issue). Thus, we have to use the library installed
154 from sources using latest stable version which works with collectd.
156 Install Apache Kafka C/C++ client library:
160 $ git clone https://github.com/edenhill/librdkafka.git ~/librdkafka
162 $ git checkout -b v0.9.5 v0.9.5
163 $ ./configure --prefix=/usr
167 Build collectd with Kafka support:
171 $ git clone https://github.com/collectd/collectd.git ~/collectd
174 $ ./configure --with-librdkafka=/usr --without-perl-bindings --enable-perl=no
175 $ make && sudo make install
177 Configure and start collectd. Create ``/opt/collectd/etc/collectd.conf``
178 collectd configuration file as following:
180 .. note:: The following collectd configuration file allows user to run VES
181 application in the guest mode. To run the VES in host mode, please follow
182 the `Configure VES in host mode`_ steps.
184 .. include:: collectd-ves-guest.conf
187 Start collectd process as a service as described in :ref:`install-collectd-as-a-service`.
189 .. Start collectd process as a service as described in `Barometer User Guide
190 <http://artifacts.opnfv.org/barometer/docs/index.html#installing-collectd-as-a-service>`_.
193 Setup VES Test Collector
194 ------------------------
196 .. note:: Test Collector setup is required only for VES application testing
199 Install dependencies:
203 $ sudo pip install jsonschema
205 Clone VES Test Collector:
209 $ git clone https://github.com/att/evel-test-collector.git ~/evel-test-collector
211 Modify VES Test Collector config file to point to existing log directory and
216 $ sed -i.back 's/^\(log_file[ ]*=[ ]*\).*/\1collector.log/' ~/evel-test-collector/config/collector.conf
217 $ sed -i.back 's/^\(schema_file[ ]*=.*\)event_format_updated.json$/\1CommonEventFormat.json/' ~/evel-test-collector/config/collector.conf
219 Start VES Test Collector:
223 $ cd ~/evel-test-collector/code/collector
224 $ nohup python ./collector.py --config ../../config/collector.conf > collector.stdout.log &
227 Setup VES application (guest mode)
228 ----------------------------------
230 Install dependencies:
234 $ sudo pip install pyyaml
236 Clone Barometer repo:
240 $ git clone https://gerrit.opnfv.org/gerrit/barometer ~/barometer
241 $ cd ~/barometer/3rd_party/collectd-ves-app/ves_app
242 $ nohup python ves_app.py --events-schema=guest.yaml --config=ves_app_config.conf > ves_app.stdout.log &
246 The above configuration is used for a localhost. The VES application can be
247 configured to use remote real VES collector and remote Kafka server. To do
248 so, the IP addresses/host names needs to be changed in ``collector.conf``
249 and ``ves_app_config.conf`` files accordingly.
252 Configure VES in host mode
253 --------------------------
255 Running the VES in host mode looks like steps described in
256 `Setup VES application (guest mode)`_ but with the following exceptions:
258 - The ``host.yaml`` configuration file should be used instead of ``guest.yaml``
259 file when VES application is running.
261 - Collectd should be running on host machine only.
263 - Addition ``libvirtd`` dependencies needs to be installed on a host where
264 collectd daemon is running. To install those dependencies, see :ref:`virt-plugin`
265 section of Barometer user guide.
267 - At least one VM instance should be up and running by hypervisor on the host.
269 - The next (minimum) configuration needs to be provided to collectd to be able
270 to generate the VES message to VES collector.
272 .. include:: collectd-ves-host.conf
275 to apply this configuration, the ``/opt/collectd/etc/collectd.conf`` file
276 needs to be modified based on example above and collectd daemon needs to
277 be restarted using the command below:
281 $ sudo systemctl restart collectd
283 .. note:: The list of the plugins can be extented depends on your needs.
286 VES application configuration description
287 -----------------------------------------
289 **Details of the Vendor Event Listener REST service**
291 REST resources are defined with respect to a ``ServerRoot``::
293 ServerRoot = https://{Domain}:{Port}/{optionalRoutingPath}
295 REST resources are of the form::
297 {ServerRoot}/eventListener/v{apiVersion}`
298 {ServerRoot}/eventListener/v{apiVersion}/{topicName}`
299 {ServerRoot}/eventListener/v{apiVersion}/eventBatch`
301 Within the VES directory (``3rd_party/collectd-ves-app/ves_app``) there is a
302 configuration file called ``ves_app.conf``. The description of the
303 configuration options are described below:
306 VES domain name. It can be IP address or hostname of VES collector
307 (default: ``127.0.0.1``)
310 VES port (default: ``30000``)
313 Used as the "optionalRoutingPath" element in the REST path (default: empty)
316 Used as the "topicName" element in the REST path (default: empty)
318 **UseHttps** *true|false*
319 Allow application to use HTTPS instead of HTTP (default: ``false``)
321 **Username** *"username"*
322 VES collector user name (default: empty)
324 **Password** *"passwd"*
325 VES collector password (default: empty)
327 **SendEventInterval** *interval*
328 This configuration option controls how often (sec) collectd data is sent to
329 Vendor Event Listener (default: ``20``)
331 **ApiVersion** *version*
332 Used as the "apiVersion" element in the REST path (default: ``5.1``)
335 Kafka Port (Default ``9092``)
337 **KafkaBroker** *host*
338 Kafka Broker domain name. It can be an IP address or hostname of local or remote server
339 (default: ``localhost``)
342 VES notification support
343 ------------------------
345 The VES application already supports YAML notification definitions but due to
346 the collectd Kafka plugin limitations, collectd notifications cannot be received
347 by the VES application. Thus, the VES notification (defined by YAML) will not be
348 generated and sent to VES collector.