1. Dependencies: install JAVA & Zookeeper.
- JAVA:
+ Ubuntu 16.04:
.. code:: bash
- $ sudo apt install default-jre
+ $ sudo apt-get install default-jre
+ $ sudo apt-get install zookeeperd
+ $ sudo apt-get install python-pip
- CentOS 7.x use:
+ CentOS:
.. code:: bash
$ sudo yum install java-1.6.0-openjdk
-
- Zookeeper:
-
- .. code:: bash
-
- $ sudo apt install zookeeperd
-
- CentOS 7.x use:
-
- .. code:: bash
-
+ $ sudo yum install python-pip
$ sudo yum install zookeeper
.. note:: You may need to add the repository that contains zookeeper.
$ sudo yum install
https://archive.cloudera.com/cdh5/one-click-install/redhat/7/x86_64/cloudera-cdh-5-0.x86_64.rpm
- CentOS 7.x start zookeeper:
+ Start zookeeper:
.. code:: bash
.. code:: bash
- $ sudo /usr/lib/zookeeper/bin/zkServer-initialize.sh
+ $ sudo /usr/lib/zookeeper/bin/zkServer-initialize.sh
No myid provided, be sure to specify it in /var/lib/zookeeper/myid if using non-standalone
- To test if Zookeeper is running as a daemon.
+2. Test if Zookeeper is running as a daemon.
.. code:: bash
$ telnet localhost 2181
Type 'ruok' & hit enter.
- Expected response is 'imok'. Zookeeper is running fine.
+ Expected response is 'imok' which means that Zookeeper is up running.
- .. note::
+3. Install Kafka
- VES doesn't work with version 0.9.4 of kafka-python.
- The recommended/tested version is 1.3.3.
+ .. note:: VES doesn't work with version 0.9.4 of kafka-python. The
+ recommended/tested version is 1.3.3.
.. code:: bash
- $ sudo yum install python-pip
$ sudo pip install kafka-python
-
-2. Download Kafka:
-
- .. code:: bash
-
- $ wget "http://www-eu.apache.org/dist/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz"
-
-3. Extract the archive:
-
- .. code:: bash
-
+ $ wget "https://archive.apache.org/dist/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz"
$ tar -xvzf kafka_2.11-0.11.0.0.tgz
+ $ sed -i -- 's/#delete.topic.enable=true/delete.topic.enable=true/' kafka_2.11-0.11.0.0/config/server.properties
+ $ sudo nohup kafka_2.11-0.11.0.0/bin/kafka-server-start.sh \
+ kafka_2.11-0.11.0.0/config/server.properties > kafka_2.11-0.11.0.0/kafka.log 2>&1 &
-4. Configure Kafka Server:
-
- .. code:: bash
-
- $ vi kafka_2.11-0.11.0.0/config/server.properties
-
- By default Kafka does not allow you to delete topics. Please uncomment:
-
- .. code:: bash
-
- delete.topic.enable=true
-
-5. Start the Kafka Server.
-
- Run 'kafka-server-start.sh' with nohup as a background process:
+ .. note:: If Kafka server fails to start, please check if the system IP
+ address is associated with the hostname in the static host lookup
+ table. If it doesn't exist, use the following command to add it.
.. code:: bash
- $ sudo nohup kafka_2.11-0.11.0.0/bin/kafka-server-start.sh \
- kafka_2.11-0.11.0.0/config/server.properties > kafka_2.11-0.11.0.0/kafka.log 2>&1 &
+ $ echo "$(ip route get 8.8.8.8 | awk '{print $NF; exit}') $HOSTNAME" | sudo tee -a /etc/hosts
-6. Test Kafka Broker Installation
+4. Test the Kafka Installation
To test if the installation worked correctly there are two scripts, producer and consumer scripts.
These will allow you to see messages pushed to broker and receive from broker.
.. code:: bash
$ kafka_2.11-0.11.0.0/bin/kafka-console-consumer.sh --zookeeper \
- localhost:2181 --topic TopicTest --from-beginning
+ localhost:2181 --topic TopicTest --from-beginning --max-messages 1 --timeout-ms 3000
Install collectd
Install development tools:
-.. code:: bash
+ Ubuntu 16.04:
+
+ .. code:: bash
+
+ $ sudo apt-get install build-essential bison autotools-dev autoconf
+ $ sudo apt-get install pkg-config flex libtool
+
+ CentOS:
- $ sudo yum group install 'Development Tools'
+ .. code:: bash
+
+ $ sudo yum group install 'Development Tools'
-.. The libkafka installed via yum pkg manager is 0.11.0 which doesn't work with
- collectd (compilation issue). Thus, we have to use the library installed
- from sources using latest stable version which works with collectd.
+.. The libkafka installed via the package manager may not work with collectd
+ (due to compilation issue). Thus, it's recommented to use the library installed
+ from sources using latest stable version of libkafka.
Install Apache Kafka C/C++ client library:
Setup VES application (guest mode)
----------------------------------
+This mode is used to collect guest VM statistics provided by collectd
+and send those metrics into the VES collector.
+
+.. figure:: ves-app-guest-mode.png
+
+ VES guest mode setup
+
Install dependencies:
.. code:: bash
Configure VES in host mode
--------------------------
+This mode is used to collect hypervisor statistics about guest VMs and to send
+those metrics into the VES collector. Also, this mode collects host statistics
+and send them as part of the guest VES message.
+
+.. figure:: ves-app-host-mode.png
+
+ VES host mode setup
+
Running the VES in host mode looks like steps described in
`Setup VES application (guest mode)`_ but with the following exceptions:
Code Review / barometer.git / commitdiff
