+++ /dev/null
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
-.. (c) <optionally add copywriters name>
-
-Low Latency Environment
-=======================
-
-Achieving low latency with the KVM4NFV project requires setting up a special
-test environment. This environment includes the BIOS settings, kernel
-configuration, kernel parameters and the run-time environment.
-
-Hardware Environment Description
---------------------------------
-
-BIOS setup plays an important role in achieving real-time latency. A collection
-of relevant settings, used on the platform where the baseline performance data
-was collected, is detailed below:
-
-CPU Features
-~~~~~~~~~~~~
-
-Some special CPU features like TSC-deadline timer, invariant TSC and Process posted
-interrupts, etc, are helpful for latency reduction.
-
-Below is the CPU information on the baseline test platform.
-::
- processor : 35
- vendor_id : GenuineIntel
- cpu family : 6
- model : 63
- model name : Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
- stepping : 2
- microcode : 0x2d
- cpu MHz : 2294.795
- cache size : 46080 KB
- physical id : 1
- siblings : 18
- core id : 27
- cpu cores : 18
- apicid : 118
- initial apicid : 118
- fpu : yes
- fpu_exception : yes
- cpuid level : 15
- wp : yes
- flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
- mca cmov pat pse36 clflush dts acpi mmx fxsr sse
- sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm
- constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc
- aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2
- ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt
- tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm arat epb
- pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase
- tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc
- cqm_occup_llcbugs
- bogomips : 4595.54
- clflush size : 64
- cache_alignment : 64
- address sizes : 46 bits physical, 48 bits virtual
- power management:
-
-CPU Topology
-~~~~~~~~~~~~
-
-NUMA topology is also important for latency reduction.
-
-Below is the CPU topology on the baseline test platform.
-::
- [nfv@otcnfv02 ~]$ lscpu
- Architecture: x86_64
- CPU op-mode(s): 32-bit, 64-bit
- Byte Order: Little Endian
- CPU(s): 36
- On-line CPU(s) list: 0-35
- Thread(s) per core: 1
- Core(s) per socket: 18
- Socket(s): 2
- NUMA node(s): 2
- Vendor ID: GenuineIntel
- CPU family: 6
- Model: 63
- Model name: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
- Stepping: 2
- CPU MHz: 2294.795
- BogoMIPS: 4595.54
- Virtualization: VT-x
- L1d cache: 32K
- L1i cache: 32K
- L2 cache: 256K
- L3 cache: 46080K
- NUMA node0 CPU(s): 0-17
- NUMA node1 CPU(s): 18-35
-
-BIOS Setup
-~~~~~~~~~~
-
-Careful BIOS setup is important in achieving real time latency. Different
-platforms have different BIOS setups, below are the important BIOS settings on
-the platform used to collect the baseline performance data.
-::
- CPU Power and Performance <Performance>
- CPU C-State <Disabled>
- C1E Autopromote <Disabled>
- Processor C3 <Disabled>
- Processor C6 <Disabled>
- Select Memory RAS <Maximum Performance>
- NUMA Optimized <Enabled>
- Cluster-on-Die <Disabled>
- Patrol Scrub <Disabled>
- Demand Scrub <Disabled>
- Correctable Error <10>
- Intel(R) Hyper-Threading <Disabled>
- Active Processor Cores <All>
- Execute Disable Bit <Enabled>
- Intel(R) Virtualization Technology <Enabled>
- Intel(R) TXT <Disabled>
- Enhanced Error Containment Mode <Disabled>
- USB Controller <Enabled>
- USB 3.0 Controller <Auto>
- Legacy USB Support <Disabled>
- Port 60/64 Emulation <Disabled>
-
-Software Environment Setup
---------------------------
-Both the host and the guest environment need to be configured properly to
-reduce latency variations. Below are some suggested kernel configurations.
-The ci/envs/ directory gives detailed implementation on how to setup the
-environment.
-
-Kernel Parameter
-~~~~~~~~~~~~~~~~
-
-Please check the default kernel configuration in the source code at:
-kernel/arch/x86/configs/opnfv.config.
-
-Below is host kernel boot line example:
-::
- isolcpus=11-15,31-35 nohz_full=11-15,31-35 rcu_nocbs=11-15,31-35 iommu=pt intel_iommu=on default_hugepagesz=1G hugepagesz=1G mce=off idle=poll intel_pstate=disable processor.max_cstate=1 pcie_asmp=off tsc=reliable
-
-Below is guest kernel boot line example
-::
- isolcpus=1 nohz_full=1 rcu_nocbs=1 mce=off idle=poll default_hugepagesz=1G hugepagesz=1G
-
-Please refer to :doc:`tunning` for more explanation.
-
-Run-time Environment Setup
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Not only are special kernel parameters needed but a special run-time
-environment is also required. Please refer to :doc:`tunning` for more
-explanation.
+++ /dev/null
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
-.. (c) <optionally add copywriters name>
-
-===============
-KVM4NFV project
-===============
-
-Welcome to KVM4NFV_ project!
-
-
-
-.. _KVM4NFV: https://wiki.opnfv.org/nfv-kvm
-
-Contents:
-
-KVM4NFV Project Description
-===========================
-
-The NFV hypervisors provide crucial functionality in the NFV Infrastructure
-(NFVI). The existing hypervisors, however, are not necessarily designed or
-targeted to meet the requirements for the NFVI, and we need to make
-collaborative efforts toward enabling the NFV features.
-
-The KVM4NFV project focuses on the KVM hypervisor to enhance it for NFV, by
-looking at the following areas
-
-+ Minimal Interrupt latency variation for data plane VNFs
- * Minimal Timing Variation for Timing correctness of real-time VNFs
- * Minimal packet latency variation for data-plane VNFs
-+ Fast live migration
-
-While these items require software development and/or specific hardware features
-there are also some adjustments that need to be made to system configuration
-information, like hardware, BIOS, OS, etc.
-
-.. toctree::
- :numbered:
- :maxdepth: 1
-
-Setup Guides
-============
-.. toctree::
- :maxdepth: 2
-
- environment-setup
- tuning
- live_migration
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+========
+Abstract
+========
+
+This document provides guidance for the configurations available in the
+Colorado release of OPNFV.
+
+The release includes four installer tools leveraging different technologies;
+Apex, Compass4nfv, Fuel and JOID, which deploy components of the platform.
+
+This document also includes the selection of tools and components including
+guidelines for how to deploy and configure the platform to an operational
+state.
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+======================
+Configuration Options
+======================
+
+OPNFV provides a variety of virtual infrastructure deployments called scenarios
+designed to host virtualised network functions (VNF's). KVM4NFV scenarios
+provide specific capabilities and/or components aimed to solve specific
+problems for the deployment of VNF's. KVM4NFV scenario includes components
+such as OpenStack,KVM etc. which includes different source components or
+configurations.
+
+KVM4NFV Scenarios
+===================
+
+Each KVM4NFV scenario provides unique features and capabilities, it is
+important to understand your target platform capabilities before installing
+and configuring. This configuration guide outlines how to install and
+configure components in order to enable the features required.
+
+.. include:: scenariomatrix.rst
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+*************************
+OPNFV Configuration Guide
+*************************
+Colorado 1.0
+------------
+
+.. toctree::
+ :maxdepth: 2
+
+ ./abstract.rst
+ ./configuration.options.render.rst
+ ./low-latency.feature.configuration.description.rst
+ ./os-nosdn-kvm-ha.description.rst
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+Introduction
+============
+
+In KVM4NFV project, we focus on the KVM hypervisor to enhance it for NFV, by
+looking at the following areas initially
+
+* Minimal Interrupt latency variation for data plane VNFs:
+ * Minimal Timing Variation for Timing correctness of real-time VNFs
+ * Minimal packet latency variation for data-plane VNFs
+* Inter-VM communication,
+* Fast live migration
+
+Configuration of Cyclictest
+===========================
+
+Cyclictest measures Latency of response to a stimulus. Achieving low latency
+with the KVM4NFV project requires setting up a special test environment.
+This environment includes the BIOS settings, kernel configuration, kernel
+parameters and the run-time environment.
+
+* For more information regarding the test environment, please visit
+ https://wiki.opnfv.org/display/kvm/KVM4NFV+Test++Environment
+ https://wiki.opnfv.org/display/kvm/Nfv-kvm-tuning
+
+Pre-configuration activities
+----------------------------
+
+Intel POD1 is currently used as OPNFV-KVM4NFV test environment. The latest
+build packages are downloaded onto Intel Pod1-jump server from artifact
+repository. Yardstick running in a ubuntu docker container on Intel Pod1-jump
+server will trigger the cyclictest.
+
+Running cyclictest through Yardstick will Configure the host(Pod1-node1), the
+guest, executes cyclictest on the guest.
+
+The following scripts are used for configuring host and guest to create a
+special test environment and achieve low latency.
+
+**host-setup0.sh**: On running this script will install latest kernel rpm
+on host and will make necessary changes as following to create special test
+environment
+
+ * Isolates CPUs from the general scheduler
+ * Stops timer ticks on isolated CPUs whenever possible
+ * Stops RCU callbacks on isolated CPUs
+ * Enables intel iommu driver and disables DMA translation for devices
+ * Sets HugeTLB pages to 1GB
+ * Disables machine check
+ * Disables clocksource verification at runtime
+
+**host-setup1.sh**: On running this script will make following test
+environment changes
+
+ * Disabling watchdogs to reduce overhead
+ * Disabling RT throttling
+ * Reroute interrupts bound to isolated CPUs to CPU 0
+ * Change the iptable so that we can ssh to the guest remotely
+
+**host-run-qemu.sh**: On running this script will launch a guest vm on host.
+ Note: download guest disk image from artifactory
+
+**guest-setup0.sh**: On running this scrcipt on guest vm will install the
+latest build kernel rpm, cyclictest and makes following configuration on
+guest vm.
+
+ * Isolates CPUs from the general scheduler
+ * Stops timer ticks on isolated CPUs whenever possible
+ * Uses polling idle loop to improve performance
+ * Disables clocksource verification at runtime
+
+**guest-setup1.sh**: On running this script on guest vm will make following
+configurations
+
+ * Disable watchdogs to reduce overhead
+ * Routes device interrupts to non-RT CPU
+ * Disables RT throttling
+
+Hardware configuration
+----------------------
+
+Currently Intel POD1 is used as test environment for kvmfornfv to execute
+cyclictest. As part of this test environment Intel pod1-jump is configured as
+jenkins slave and all the latest build artifacts are downloaded on to it.
+Intel pod1-node1 is the host on which a guest vm will be launched as a part of
+running cylictest through yardstick.
+
+* For more information regarding hardware configuration, please visit
+ https://wiki.opnfv.org/display/pharos/Intel+Pod1
+ https://build.opnfv.org/ci/computer/intel-pod1/
+ http://artifacts.opnfv.org/octopus/brahmaputra/docs/octopus_docs/opnfv-jenkins-slave-connection.html
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+
+.. http://creativecommons.org/licenses/by/4.0
+
+
+Introduction
+============
+
+.. In this section explain the purpose of the scenario and the
+ types of capabilities provided
+
+The purpose of os-nosdn-kvm-ha scenario testing is to test the
+High Availability deployment and configuration of OPNFV software suite
+with OpenStack and without SDN software. This OPNFV software suite
+includes OPNFV KVM4NFV latest software packages for Linux Kernel and
+QEMU patches for achieving low latency. High Availability feature is achieved
+by deploying OpenStack multi-node setup with 3 controllers and 2 computes nodes
+
+KVM4NFV packages will be installed on compute nodes as part of deployment.
+This scenario testcase deployment is happening on multi-node by using
+OPNFV Fuel deployer.
+
+Scenario Components and Composition
+===================================
+.. In this section describe the unique components that make up the scenario,
+.. what each component provides and why it has been included in order
+.. to communicate to the user the capabilities available in this scenario.
+
+This scenario deploys the High Availability OPNFV Cloud based on the
+configurations provided in ha_nfv-kvm_heat_ceilometer_scenario.yaml.
+This yaml file contains following configurations and is passed as an
+argument to deploy.py script
+
+* scenario.yaml:This configuration file defines translation between a
+ short deployment scenario name(os-nosdn-kvm-ha) and an actual deployment
+ scenario configuration file(ha_nfv-kvm_heat_ceilometer_scenario.yaml)
+
+* deployment-scenario-metadata:Contains the configuration metadata like
+ title,version,created,comment.
+
+* stack-extensions:Stack extentions are opnfv added value features in form
+ of a fuel-plugin.Plugins listed in stack extensions are enabled and
+ configured.
+
+* dea-override-config: Used to configure the HA mode,network segmentation
+ types and role to node assignments.These configurations overrides
+ corresponding keys in the dea_base.yaml and dea_pod_override.yaml.
+ These keys are used to deploy multiple nodes(3 controllers,2 computes)
+ as mention below.
+
+ * **Node 1**: This node has MongoDB and Controller roles. The controller
+ node runs the Identity service, Image Service, management portions of
+ Compute and Networking, Networking plug-in and the dashboard. The
+ Telemetry service which was designed to support billing systems for
+ OpenStack cloud resources uses a NoSQL database to store information.
+ The database typically runs on the controller node.
+
+ * **Node 2**: This node has Controller and Ceph-osd roles. Ceph is a
+ massively scalable, open source, distributed storage system. It is
+ comprised of an object store, block store and a POSIX-compliant distributed
+ file system. Enabling Ceph, configures Nova to store ephemeral volumes in
+ RBD, configures Glance to use the Ceph RBD backend to store images,
+ configures Cinder to store volumes in Ceph RBD images and configures the
+ default number of object replicas in Ceph.
+
+ * **Node 3**: This node has Controller role in order to achieve high
+ availability.
+
+ * **Node 4**: This node has Compute role. The compute node runs the
+ hypervisor portion of Compute that operates tenant virtual machines
+ or instances. By default, Compute uses KVM as the hypervisor.
+
+ * **Node 5**: This node has compute role.
+
+* dha-override-config:Provides information about the VM definition and
+ Network config for virtual deployment.These configurations overrides
+ the pod dha definition and points to the controller,compute and
+ fuel definition files.
+
+* os-nosdn-kvm-ha scenario is successful when all the 5 Nodes are accessible,
+ up and running
+
+Scenario Usage Overview
+=======================
+.. Provide a brief overview on how to use the scenario and the features available to the
+.. user. This should be an "introduction" to the userguide document, and explicitly link to it,
+.. where the specifics of the features are covered including examples and API's
+
+* The high availability feature can be acheived by executing deploy.py with
+ ha_nfv-kvm_heat_ceilometer_scenario.yaml as an argument.
+* Install Fuel Master and deploy OPNFV Cloud from scratch on Hardware
+ Environment:
+
+ -Example:
+
+ sudo python deploy.py -iso ~/ISO/opnfv.iso -dea ~/CONF/hardware/dea.yaml -dha ~/CONF/hardware/dha.yaml -s /mnt/images -b pxebr -log ~/Deployment-888.log.tar.gz
+
+* Install Fuel Master and deploy OPNFV Cloud from scratch on Virtual
+ Environment:
+
+ -Example:
+
+ sudo python deploy.py -iso ~/ISO/opnfv.iso -dea ~/CONF/virtual/dea.yaml -dha ~/CONF/virtual/dha.yaml -s /mnt/images -log ~/Deployment-888.log.tar.gz
+
+* os-nosdn-kvm-ha scenario can be executed from the jenkins project
+ "fuel-os-nosdn-kvm-ha-baremetal-daily-master"
+* This scenario provides the High Availability feature by deploying
+ 3 controller,2 compute nodes and checking if all the 5 nodes
+ are accessible(IP,up & running).
+* Test Scenario is passed if deployment is successful and all 5 nodes have
+ accessibility (IP , up & running).
+* Observed that scenario is not running any testcase on top of deployment.
+
+Known Limitations, Issues and Workarounds
+=========================================
+.. Explain any known limitations here.
+
+* Test scenario os-nosdn-kvm-ha result is not stable. After node reboot
+ triggered by kvm plugin, sometimes puppet agent (mcollective) is not
+ responding with in the given time.
+
+References
+==========
+
+For more information on the OPNFV Colorado release, please visit
+http://www.opnfv.org/colorado
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+
+.. http://creativecommons.org/licenses/by/4.0
+
+Scenarios are implemented as deployable compositions through integration with an installation tool.
+OPNFV supports multiple installation tools and for any given release not all tools will support all
+scenarios. While our target is to establish parity across the installation tools to ensure they
+can provide all scenarios, the practical challenge of achieving that goal for any given feature and
+release results in some disparity.
+
+Colorado scenario overeview
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following table provides an overview of the installation tools and available scenario's
+in the Colorado release of OPNFV.
+
+Scenario status is indicated by a weather pattern icon. All scenarios listed with
+a weather pattern are possible to deploy and run in your environment or a Pharos lab,
+however they may have known limitations or issues as indicated by the icon.
+
+Weather pattern icon legend:
+
++---------------------------------------------+----------------------------------------------------------+
+| Weather Icon | Scenario Status |
++=============================================+==========================================================+
+| .. image:: ../images/weather-clear.jpg | Stable, no known issues |
++---------------------------------------------+----------------------------------------------------------+
+| .. image:: ../images/weather-few-clouds.jpg | Stable, documented limitations |
++---------------------------------------------+----------------------------------------------------------+
+| .. image:: ../images/weather-overcast.jpg | Deployable, stability or feature limitations |
++---------------------------------------------+----------------------------------------------------------+
+| .. image:: ../images/weather-dash.jpg | Not deployed with this installer |
++---------------------------------------------+----------------------------------------------------------+
+
+Scenarios that are not yet in a state of "Stable, no known issues" will continue to be stabilised
+and updates will be made on the stable/colorado branch. While we intend that all Colorado
+scenarios should be stable it is worth checking regularly to see the current status. Due to
+our dependency on upstream communities and code some issues may not be resolved prior to the D release.
+
+Scenario Naming
+^^^^^^^^^^^^^^^
+
+In OPNFV scenarios are identified by short scenario names, these names follow a scheme that
+identifies the key components and behaviours of the scenario. The rules for scenario naming are as follows:
+
+ os-[controller]-[feature]-[mode]-[option]
+
+Details of the fields are
+ * os: mandatory
+
+ * Refers to the platform type used
+ * possible value: os (OpenStack)
+
+* [controller]: mandatory
+
+ * Refers to the SDN controller integrated in the platform
+ * example values: nosdn, ocl, odl, onos
+
+ * [feature]: mandatory
+
+ * Refers to the feature projects supported by the scenario
+ * example values: nofeature, kvm, ovs, sfc
+
+ * [mode]: mandatory
+
+ * Refers to the deployment type, which may include for instance high availability
+ * possible values: ha, noha
+
+ * [option]: optional
+
+ * Used for the scenarios those do not fit into naming scheme.
+ * The optional field in the short scenario name should not be included if there is no optional scenario.
+
+Some examples of supported scenario names are:
+
+ * os-nosdn-kvm-noha
+
+ * This is an OpenStack based deployment using neutron including the OPNFV enhanced KVM hypervisor
+
+ * os-onos-nofeature-ha
+
+ * This is an OpenStack deployment in high availability mode including ONOS as the SDN controller
+
+ * os-odl_l2-sfc
+
+ * This is an OpenStack deployment using OpenDaylight and OVS enabled with SFC features
+
+Installing your scenario
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+There are two main methods of deploying your target scenario, one method is to follow this guide which will
+walk you through the process of deploying to your hardware using scripts or ISO images, the other method is
+to set up a Jenkins slave and connect your infrastructure to the OPNFV Jenkins master.
+
+For the purposes of evaluation and development a number of Colorado scenarios are able to be deployed
+virtually to mitigate the requirements on physical infrastructure. Details and instructions on performing
+virtual deployments can be found in the installer specific installation instructions.
+
+To set up a Jenkins slave for automated deployment to your lab, refer to the `Jenkins slave connect guide.
+<http://artifacts.opnfv.org/brahmaputra.1.0/docs/opnfv-jenkins-slave-connection.brahmaputra.1.0.html>`_
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+================
+KVMFORNFV Design
+================
+
+.. toctree::
+ :numbered:
+ :maxdepth: 3
+
+ kvmfornfv_design.rst
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+============
+Introduction
+============
+
+**Purpose**:
+
+ This document provides an overview of the areas that can be addressed to
+ enhance the KVM Hypervisor for NFV. It is intended to capture and convey the
+ significant changes which have been made on the KVM Hypervisor.
+
+
+===================
+Project description
+===================
+
+The NFV hypervisors provide crucial functionality in the NFV
+Infrastructure(NFVI).The existing hypervisors, however, are not necessarily
+designed or targeted to meet the requirements for the NFVI.
+
+This design focuses on the enhancement of following area for KVM Hypervisor
+
+* Minimal Interrupt latency variation for data plane VNFs:
+ * Minimal Timing Variation for Timing correctness of real-time VNFs
+ * Minimal packet latency variation for data-plane VNFs
+* Fast live migration
+
+While these items require software development and/or specific hardware features
+there are also some adjustments that need to be made to system configuration
+information, like hardware, BIOS, OS, etc.
+
+**Minimal Interrupt latency variation for data plane VNFs**
+
+Processing performance and latency depend on a number of factors, including
+the CPUs (frequency, power management features, etc.), micro-architectural
+resources, the cache hierarchy and sizes, memory (and hierarchy, such as NUMA)
+and speed, inter-connects, I/O and I/O NUMA, devices, etc.
+
+There are two separate types of latencies to minimize:
+
+ 1. Minimal Timing Variation for Timing correctness of real-time
+ VNFs – timing correctness for scheduling operations(such as Radio scheduling)
+ 2. Minimal packet latency variation for data-plane VNFs – packet delay
+ variation, which applies to packet processing.
+
+For a VM, interrupt latency (time between arrival of H/W interrupt and
+invocation of the interrupt handler in the VM), for example, can be either of
+the above or both, depending on the type of the device. Interrupt latency with
+a (virtual) timer can cause timing correctness issues with real-time VNFs even
+if they only use polling for packet processing.
+
+We assume that the VNFs are implemented properly to minimize interrupt latency
+variation within the VMs, but we have additional causes of latency variation
+on KVM:
+
+ - Asynchronous (e.g. external interrupts) and synchronous(e.g. instructions)
+ VM exits and handling in KVM (and kernel routines called), which may have
+ loops and spin locks
+ - Interrupt handling in the host Linux and KVM, scheduling and virtual
+ interrupt delivery to VNFs
+ - Potential VM exit (e.g. EOI) in the interrupt service routines in VNFs
+ - Exit to the user-level (e.g. QEMU)
+
+.. Figure:: kvm1.png
+
+=====================
+Design Considerations
+=====================
+
+The latency variation and jitters can be minimized with the below
+steps (with some in parallel):
+
+ 1. Statically and exclusively assign hardware resources
+ (CPUs, memory, caches,) to the VNFs.
+
+ 2. Pre-allocate huge pages (e.g. 1 GB/2MB pages) and guest-to-host mapping,
+ e.g. EPT (Extended Page Table) page tables, to minimize or mitigate
+ latency from misses in caches,
+
+ 3. Use the host Linux configured for hard real-time and packet latency,
+ Check the set of virtual devices used by the VMs to optimize or
+ eliminate virtualization overhead if applicable
+
+ 4. Use advanced hardware virtualization features that can reduce or
+ eliminate VM exits, if present, and
+
+ 5. Inspect the code paths in KVM and associated kernel services to
+ eliminate code that can cause latencies (e.g. loops and spin locks).
+
+ 6. Measure latencies intensively. We leverage the existing testing methods.
+ OSADL, for example, defines industry tests for timing correctness.
+
+====================
+Goals and Guidelines
+====================
+
+The output of this project will provide :
+
+ 1. A list of the performance goals, which will be obtained by the
+ OPNFV members (as described above)
+
+ 2. A set of comprehensive instructions for the system configurations
+ (hardware features, BIOS setup, kernel parameters, VM configuration,
+ options to QEMU/KVM, etc.)
+
+ 3. The above features to the upstream of Linux, the real-time patch
+ set, KVM, QEMU, libvirt, and
+
+ 4. Performance and interrupt latency measurement tools
+
+=========
+Test plan
+=========
+
+The tests that need to be conducted to make sure that all components from OPNFV
+meet the requirement are mentioned below:
+
+**Timer test**:This test utilize the cyclictest
+(https://rt.wiki.kernel.org/index.php/Cyclictest) to test the guest timer
+latency (the latency from the time that the guest timer should be triggered
+to the time the guest timer is really triggered).
+
+.. Figure:: TimerTest.png
+
+**Device Interrupt Test**:A device on the hardware platform trigger interrupt
+every one ms and the device interrupt will be delivered to the VNF. This test
+cover the latency from the interrupt happened on the hardware to the time the
+interrupt handled in the VNF.
+
+.. Figure:: DeviceInterruptTest.png
+
+**Packet forwarding (DPDK OVS)**:A packet is sent from TC (Traffic Generator)
+to a VNF. The VNF, after processing the packet, forwards the packet to another
+NIC and in the end the packet is received by the traffic generator. The test
+check the latency from the packet is sent out by the TC to the time the packet
+is received by the TC.
+
+.. Figure:: PacketforwardingDPDK_OVS.png
+
+**Packet Forwarding (SR-IOV)**:This test is similar to Packet Forwarding
+(DPDK OVS). However, instead of using virtio NIC devices on the guest,
+a PCI NIC or a PCI VF NIC is assigned to the guest for network acess.
+
+**Bare-metal Packet Forwarding**:This is used to compare with the above
+packet forwarding scenario.
+
+.. Figure:: Bare-metalPacketForwarding.png
+
+=========
+Reference
+=========
+
+https://wiki.opnfv.org/display/kvm/
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+
+.. http://creativecommons.org/licenses/by/4.0
+
+**************
+OPNFV Glossary
+**************
+Colorado 1.0
+------------
+
+========
+Contents
+========
+
+This glossary provides a common definition of phrases and words commonly used
+in OPNFV.
+
+--------
+
+A
+-
+
+Arno
+
+ A river running through Tuscany and the name of the first OPNFV release.
+
+API
+
+ Application Programming Interface
+
+AVX2
+
+ Advanced Vector Extensions 2 is an instruction set extension for x86.
+
+
+--------
+
+B
+-
+
+Brahmaputra
+
+ A river running through Asia and the name of the Second OPNFV release.
+
+Bios
+
+ Basic Input/Output System
+
+Builds
+
+ Build in Jenkins is a version of a program.
+
+Bogomips
+
+ Bogomips is the number of million times per second a processor can do
+ absolutely nothing.
+
+--------
+
+C
+-
+
+CAT
+
+ Cache Automation Technology
+
+CentOS
+
+ Community Enterprise Operating System is a Linux distribution
+
+CICD
+
+ Continuous Integration and Continuous Deployment
+
+CLI
+
+ Command Line Interface
+
+Colorado
+
+ A river in Argentina and the name of the Third OPNFV release.
+
+Compute
+
+ Compute is an OpenStack service which offers many configuration options
+ which may be deployment specific.
+
+Console
+
+ Console is display screen.
+
+CPU
+ Central Processing Unit
+
+--------
+
+D
+-
+
+Data plane
+
+ The data plane is the part of a network that carries user traffic.
+
+Debian/deb
+
+ Debian is a Unix-like computer operating system that is composed entirely of
+ free software.
+
+Docs
+
+ Documentation/documents
+
+DPDK
+
+ Data Plane Development Kit
+
+DPI
+
+ Deep Packet Inspection
+
+DSCP
+
+ Differentiated Services Code Point
+
+--------
+
+F
+-
+
+Flavors
+
+ Flavors are templates used to define VM configurations.
+
+Fuel
+
+ Provides an intuitive, GUI-driven experience for deployment and management of OpenStack
+
+--------
+
+H
+-
+
+Horizon
+
+ Horizon is an OpenStack service which serves as an UI.
+
+Hypervisor
+
+ A hypervisor, also called a virtual machine manager, is a program that allows
+ multiple operating systems to share a single hardware host.
+
+--------
+
+I
+-
+
+IGMP
+
+ Internet Group Management Protocol
+
+IOMMU
+
+ Input-Output Memory Management Unit
+
+IOPS
+
+ Input/Output Operations Per Second
+
+IRQ
+
+ Interrupt ReQuest is an interrupt request sent from the hardware level to
+ the CPU.
+
+IRQ affinity
+
+ IRQ affinity is the set of CPU cores that can service that interrupt.
+
+--------
+
+J
+-
+
+Jenkins
+
+ Jenkins is an open source continuous integration tool written in Java.
+
+JIRA
+
+ JIRA is a bug tracking software.
+
+Jitter
+
+ Time difference in packet inter-arrival time to their destination can be called jitter.
+
+JumpHost
+
+ A jump host or jump server or jumpbox is a computer on a network typically
+ used to manage devices in a separate security zone.
+
+--------
+
+K
+-
+
+Kernel
+
+ The kernel is a computer program that constitutes the central core of a
+ computer's operating system.
+
+--------
+
+L
+-
+
+Latency
+
+ The amount of time it takes a packet to travel from source to destination is
+ Latency.
+
+libvirt
+
+ libvirt is an open source API, daemon and management tool for managing
+ platform virtualization.
+
+--------
+
+M
+-
+
+Migration
+
+ Migration is the process of moving from the use of one operating environment
+ to another operating environment.
+
+--------
+
+N
+-
+
+NFV
+
+ Network Functions Virtualisation, an industry initiative to leverage
+ virtualisation technologies in carrier networks.
+
+NFVI
+
+ Network Function Virtualization Infrastructure
+
+NIC
+
+ Network Interface Controller
+
+NUMA
+
+ Non-Uniform Memory Access
+
+--------
+
+O
+-
+
+OPNFV
+
+ Open Platform for NFV, an open source project developing an NFV reference
+ platform and features.
+
+--------
+
+P
+-
+
+Pharos
+
+ Is a lighthouse and is a project deals with developing an OPNFV lab
+ infrastructure that is geographically and technically diverse.
+
+Pipeline
+
+ A suite of plugins in Jenkins that lets you orchestrate automation.
+
+Platform
+
+ OPNFV provides an open source platform for deploying NFV solutions that
+ leverages investments from a community of developers and solution providers.
+
+Pools
+
+ A Pool is a set of resources that are kept ready to use, rather than acquired
+ on use and released afterwards.
+
+--------
+
+Q
+-
+
+Qemu
+
+ QEMU is a free and open-source hosted hypervisor that performs hardware
+ virtualization.
+
+--------
+
+R
+-
+
+RDMA
+
+ Remote Direct Memory Access (RDMA)
+
+Rest-Api
+
+ REST (REpresentational State Transfer) is an architectural style, and an
+ approach to communications that is often used in the development of web
+ services
+
+--------
+
+S
+-
+
+Scaling
+
+ Refers to altering the size.
+
+Slave
+
+ Works with/for master.where master has unidirectional control over one or
+ more other devices.
+
+SR-IOV
+
+ Single root IO- Virtualization.
+
+Spin locks
+
+ A spinlock is a lock which causes a thread trying to acquire it to simply
+ wait in a loop while repeatedly checking if the lock is available.
+
+Storage
+
+ Refers to computer components which store some data.
+
+--------
+
+T
+-
+
+Tenant
+
+ A Tenant is a group of users who share a common access with specific
+ privileges to the software instance.
+
+Tickless
+
+ A tickless kernel is an operating system kernel in which timer interrupts
+ do not occur at regular intervals, but are only delivered as required.
+
+TSC
+
+ Technical Steering Committee
+
+--------
+
+V
+-
+
+VLAN
+
+ A virtual local area network, typically an isolated ethernet network.
+
+VM
+
+ Virtual machine, an emulation in software of a computer system.
+
+VNF
+
+ Virtual network function, typically a networking application or function
+ running in a virtual environment.
+
+--------
+
+X
+-
+
+XBZRLE
+
+ Helps to reduce the network traffic by just sending the updated data
+
+--------
+
+Y
+-
+
+Yardstick
+
+ Yardstick is an infrastructure verification. It is an OPNFV testing project.
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+
+.. http://creativecommons.org/licenses/by/4.0
+
+This document will give the user instructions on how to deploy available
+KVM4NFV CICD build scenario verfied for the Colorado release of the OPNFV
+platform.
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+
+.. http://creativecommons.org/licenses/by/4.0
+
+**********************
+Installation procedure
+**********************
+Colorado 1.0
+------------
+
+.. toctree::
+ :numbered:
+ :maxdepth: 2
+
+ abstract.rst
+ kvm4nfv-cicd.installation.instruction.rst
+ kvm4nfv-cicd.release.notes.rst
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+
+.. http://creativecommons.org/licenses/by/4.0
+
+=====================================
+KVM4NFV CICD Installation Instruction
+=====================================
+
+Preparing the installation
+--------------------------
+
+The OPNFV project- KVM4NFV (https://gerrit.opnfv.org/gerrit/kvmfornfv.git) is
+cloned first, to make the build scripts for Qemu & Kernel, Rpms and Debians
+available.
+
+HW requirements
+---------------
+
+These build scripts are triggered on the Jenkins-Slave build server. Currently
+Intel POD1 is used as test environment for kvmfornfv to execute cyclictest. As
+part of this test environment Intel pod1-jump is configured as jenkins slave
+and all the latest build artifacts are downloaded on to it. Intel pod1-node1
+is the host on which a guest vm will be launched as a part of running cylictest
+through yardstick.
+
+Build instructions
+------------------
+
+Builds are possible for the following packages-
+
+**kvmfornfv source code**- The ./ci/build.sh is the main script used to trigger
+the Rpms (on 'centos') and Debians (on 'ubuntu') builds in this case.
+
+* How to build Kernel/Qemu Rpms- To build rpm packages, build.sh script is run
+ with -p and -o option (i.e. if -p package option is passed as "centos" or in
+ default case). Example: sh ./ci/build.sh -p centos -o build_output
+
+* How to build Kernel/Qemu Debians- To build debian packages, build.sh script
+ is run with -p and -o option (i.e. if -p package option is passed as
+ "ubuntu"). Example: sh ./ci/build.sh -p ubuntu -o build_output
+
+* How to build all Kernel & Qemu, Rpms & Debians- To build both debian and rpm
+ packages, build.sh script is run with -p and -o option (i.e. if -p package
+ option is passed as "both"). Example: sh ./ci/build.sh -p both -o build_output
+
+Installation instructions
+-------------------------
+
+Installation can be done in the following ways-
+
+**1. From kvmfornfv source code**-
+The build packages that are prepared in the above section, are installed
+differently depending on the platform.
+
+Please visit the links for each-
+
+* Centos : https://www.centos.org/docs/5/html/Deployment_Guide-en-US/s1-rpm-using.html
+* Ubuntu : https://help.ubuntu.com/community/InstallingSoftware
+
+Test the built packages by executing the scripts present in ci/envs for
+configuring host and guest respectively. Once the setup is in place, cyclictest
+is performed via yardtick, using ./ci/test_kvmfornfv.sh
+
+**2. Using Fuel installer**-
+
+* Please refer to the document present at /fuel-plugin/README.md
+
+Post-installation activities
+----------------------------
+
+After the rpm and debian builds are deployed successfully on the host-guest and
+give the expected cyclictest results, jenkins gives +1 to indicate the
+completion of verify process. Thereafter, the releng executes the merge process
+to merge this code into parent repository.
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+
+.. http://creativecommons.org/licenses/by/4.0
+
+=============================
+Release Note for KVM4NFV CICD
+=============================
+
+
+Abstract
+========
+
+This document contains the release notes for the Colorado release of
+OPNFV when using KVM4NFV CICD process.
+
+Introduction
+============
+
+Provide a brief introduction of how this configuration is used in OPNFV release
+using KVM4VFV CICD as scenario.
+
+Be sure to reference your scenario installation instruction.
+
+Release Data
+============
+
++--------------------------------------+--------------------------------------+
+| **Project** | NFV Hypervisors-KVM |
+| | |
++--------------------------------------+--------------------------------------+
+| **Repo/tag** | kvmfornfv |
+| | |
++--------------------------------------+--------------------------------------+
+| **Release designation** | |
+| | |
++--------------------------------------+--------------------------------------+
+| **Release date** | |
+| | |
++--------------------------------------+--------------------------------------+
+| **Purpose of the delivery** | Automate the KVM4VFV CICD scenario |
+| | |
++--------------------------------------+--------------------------------------+
+
+Deliverables
+------------
+
+Software deliverables
+~~~~~~~~~~~~~~~~~~~~~
+Kernel and Qemu- RPM and Debian build packages
+
+Documentation deliverables
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+- KVM4NFV CICD process documentation available under <project>/docs/ under
+ various categories.
+
+Version change
+--------------
+.. This section describes the changes made since the last version of this
+.. document.
+
+Module version change
+~~~~~~~~~~~~~~~~~~~~~
+- Build scripts made available for Kernel rpm, Kernel deb, Qemu rpm, Qemu
+ deb packages.
+- Releng scripts made available to trigger these kvm4nfv build scripts for
+ automating complete CICD process.
+
+Document version change
+~~~~~~~~~~~~~~~~~~~~~~~
+The following documents are added-
+- configurationguide
+- instalationprocedure
+- userguide
+- overview
+- glossary
+- releasenotes
+
+Reason for new version
+----------------------
+
+Feature additions
+~~~~~~~~~~~~~~~~~
+
++--------------------------------------+--------------------------------------+
+| **JIRA REFERENCE** | **SLOGAN** |
+| | |
++--------------------------------------+--------------------------------------+
+| JIRA: | NFV Hypervisors-KVMKVMFORNFV-34 |
+| | |
++--------------------------------------+--------------------------------------+
+| JIRA: | NFV Hypervisors-KVMKVMFORNFV-34 |
+| | |
++--------------------------------------+--------------------------------------+
+
+Bug corrections
+~~~~~~~~~~~~~~~
+
+**JIRA TICKETS:**
+
++--------------------------------------+--------------------------------------+
+| **JIRA REFERENCE** | **SLOGAN** |
+| | |
++--------------------------------------+--------------------------------------+
+| JIRA: | |
+| | |
++--------------------------------------+--------------------------------------+
+
+
+Known Limitations, Issues and Workarounds
+=========================================
+
+System Limitations
+------------------
+
+Known issues
+------------
+
+**JIRA TICKETS:**
+
++--------------------------------------+--------------------------------------+
+| **JIRA REFERENCE** | **SLOGAN** |
+| | |
++--------------------------------------+--------------------------------------+
+| JIRA: | |
++--------------------------------------+--------------------------------------+
+| JIRA: | |
++--------------------------------------+--------------------------------------+
+
+
+Workarounds
+-----------
+See JIRA: <link>
+
+
+References
+==========
+For more information on the OPNFV Brahmaputra release, please visit
+http://www.opnfv.org/brahmaputra
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+
+.. http://creativecommons.org/licenses/by/4.0
+
+=============================
+KMV4MFV CICD Project Overview
+=============================
+
+The detailed understanding of this project is organized into different sections-
+
+* userguide - This provides the required technical assistance to the user, in
+ using the KVM4NFV CICD process.
+* installationprocedure- This will give the user instructions on how to deploy
+ available KVM4NFV CICD build scenario.
+* configurationguide- This provides guidance for configuring KVM4NFV
+ environment, even with the use of specific installer tools for deploying some
+ components, available in the Colorado release of OPNFV.
+* requirements- This includes the introduction of KVM4NFV CICD project,
+ specifications of how the project should work, and constraints placed upon
+ its execution.
+* design- This includes the parameters or design considerations taken into
+ account for achieving minimal interrupt latency for the data VNFs.
+* releasenotes- This describes a brief summary of recent changes, enhancements
+ and bug fixes in the KVM4NFV project.
+* glossary- It includes the definition of terms, used in the KVM4NFV project
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+=========================
+KVM4NFV CICD Release Note
+=========================
+
+.. toctree::
+ :maxdepth: 2
+
+ release-notes
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+
+.. http://creativecommons.org/licenses/by/4.0
+
+=====================================================
+OPNFV Release Note for "Colorado release" - KVMFORNFV
+=====================================================
+
+.. _Kvmfornfv: https://wiki.opnfv.org/display/kvm/
+
+
+Abstract
+========
+
+This document provides the release notes for Colorado release of KVMFORNFV.
+
+License
+=======
+
+KVMFORNFV is licensed under a Creative Commons Attribution 4.0 International
+License.You should have received a copy of the license along with this. If not,
+see <http://creativecommons.org/licenses/by/4.0/>.
+
+
+**Contents**
+
+1 Version History
+
+2 Important notes
+
+3 Summary
+
+4 Delivery Data
+
+5 References
+
+1 Version history
+===================
+
++--------------------+--------------------+--------------------+--------------------+
+| **Date** | **Ver.** | **Author** | **Comment** |
+| | | | |
++--------------------+--------------------+--------------------+--------------------+
+|2016-08-22 | 0.1.0 | | Colorado release |
+| | | | |
++--------------------+--------------------+--------------------+--------------------+
+
+2 Important notes
+===================
+
+The software delivered in the OPNFV KVMFORNFV_ Project, comprises the
+*ci*, the *kvmfornfv test cases*.
+
+The *KVMFORNFV* framework depends on the *Fuel* installer.
+
+
+3 Summary
+===========
+
+This Colorado release provides *KVMFORNFV* as a framework to enhance the
+KVM Hypervisor for NFV and OPNFV scenario testing, automated in the OPNFV
+CI pipeline, including:
+
+* Documentation created
+
+ * User Guide
+
+ * Configuration Guide
+
+ * Installation Procedure
+
+ * Release notes (this document)
+
+* KVMFORNFV source code
+
+* Cyclictests for KVMFORNFV
+
+For Colorado release, the KVMFORNFV uses for the following:
+
+* Automation of building the Kernel and qemu RPM's or debians
+
+* Executing the Cyclictests to check the latency
+
+* os-sdn-kvm-ha Scenario testing for High Availability Configuration using
+ Fuel Installer
+
+The *KVMFORNFV framework* is developed in the OPNFV community, by the
+KVMFORNFV_ team.
+
+4 Release Data
+================
+
++--------------------------------------+--------------------------------------+
+| **Project** | NFV Hypervisors-KVM |
+| | |
++--------------------------------------+--------------------------------------+
+| **Repo/commit-ID** | kvmfornfv |
+| | |
++--------------------------------------+--------------------------------------+
+| **Release designation** | Colorado |
+| | |
++--------------------------------------+--------------------------------------+
+| **Release date** | 2016-08-22 |
+| | |
++--------------------------------------+--------------------------------------+
+| **Purpose of the delivery** | OPNFV Colorado Releases |
+| | |
++--------------------------------------+--------------------------------------+
+
+4.1 Version change
+------------------
+
+4.1.1 Module version changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+This is the first tracked release of KVMFORNFV
+
+
+4.1.2 Document version changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+This is the initial version of the KVMFORNFV framework in OPNFV.
+
+4.2 Reason for version
+----------------------
+
+4.2.1 Feature additions
+~~~~~~~~~~~~~~~~~~~~~~~
+
++--------------------------------------+--------------------------------------+
+| **JIRA REFERENCE** | **SLOGAN** |
+| | |
++--------------------------------------+--------------------------------------+
+| JIRA: | NFV Hypervisors-KVMKVMFORNFV-34 |
+| | |
++--------------------------------------+--------------------------------------+
+| JIRA: | NFV Hypervisors-KVMKVMFORNFV-34 |
+| | |
++--------------------------------------+--------------------------------------+
+
+4.2.2 Bug corrections
+~~~~~~~~~~~~~~~~~~~~~
+
+Initial Release
+
+4.3 Deliverables
+----------------
+
+4.3.1 Software deliverables
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+KVMFORNFV framework source code <Colorado>
+
+4.3.2 Documentation deliverables
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The below documents are delivered for Colorado KVMFORNFV Release:
+
+ * User Guide
+
+ * Configuration Guide
+
+ * Installation Procedure
+
+ * Overview
+
+ * Release notes (this document)
+
+ * Glossary
+
+
+5 References
+=============
+
+For more information on the KVMFORNFV Colorado release, please see:
+
+https://wiki.opnfv.org/display/kvm/
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+================
+KVMFORNFV Design
+================
+
+.. toctree::
+ :numbered:
+ :maxdepth: 3
+
+ kvmfornfv_requirements.rst
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Intel Corporation, AT&T and others.
+
+============
+Introduction
+============
+The NFV hypervisors provide crucial functionality in the NFV
+Infrastructure(NFVI).The existing hypervisors, however, are not necessarily
+designed or targeted to meet the requirements for the NFVI.
+
+This document specifies the list of requirements that need to be met as part
+of this "NFV Hypervisors-KVM" project in Colorado release.
+
+As part of this project we need to make collaborative efforts towards enabling
+the NFV features.
+
+=================
+Scope and Purpose
+=================
+
+The main purpose of this project is to enhance the KVM hypervisor for NFV, by
+looking at the following areas initially:
+
+* Minimal Interrupt latency variation for data plane VNFs:
+ * Minimal Timing Variation for Timing correctness of real-time VNFs
+ * Minimal packet latency variation for data-plane VNFs
+* Inter-VM communication
+* Fast live migration
+
+The output of this project would be list of the performance goals,comprehensive
+instructions for the system configurations,tools to measure Performance and
+interrupt latency.
+
+===========================
+Methods and Instrumentation
+===========================
+
+The above areas would require software development and/or specific hardware
+features, and some need just configurations information for the system
+(hardware, BIOS, OS, etc.).
+
+A right configuration is critical for improving the NFV performance/latency.
+Even working on the same code base, different configurations can make
+completely different performance/latency result.
+Configurations that can be made as part of this project to tune a specific
+scenario are:
+
+ 1. **Platform Configuration** : Some hardware features like Power management,
+ Hyper-Threading,Legacy USB Support/Port 60/64 Emulation,SMI can be configured.
+ 2. **Operating System Configuration** : Some configuration features like CPU
+ isolation,Memory allocation,IRQ affinity,Device assignment for VM,Tickless,
+ TSC,Idle,_RCU_NOCB_,Disable the RT throttling,NUMA can be configured.
+ 3. **Performance/Latency Tuning** : Application level configurations like
+ timers,Making vfio MSI interrupt as non-threaded,Cache Allocation
+ Technology(CAT) enabling can be tuned to improve the NFV
+ performance/latency.
+
+=====================
+Features to be tested
+=====================
+
+The tests that need to be conducted to make sure that latency is addressed are:
+1. Timer test
+2. Device Interrupt Test
+3. Packet forwarding (DPDK OVS)
+4. Packet Forwarding (SR-IOV)
+5. Bare-metal Packet Forwarding
+
+============
+Dependencies
+============
+
+1. OPNFV Project: “Characterize vSwitch Performance for Telco NFV Use Cases”
+ (VSPERF) for performance evaluation of ivshmem vs. vhost-user.
+2. OPNFV Project: “Pharos” for Test Bed Infrastructure, and possibly
+ “Yardstick” for infrastructure verification.
+3. There are currently no similar projects underway in OPNFV or in an upstream
+ project
+4. The relevant upstream project to be influenced here is QEMU/KVM and
+ libvirt.
+5. In terms of HW dependencies, the aim is to use standard IA Server hardware
+ for this project, as provided by OPNFV Pharos.
+
+=========
+Reference
+=========
+
+https://wiki.opnfv.org/display/kvm/
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+
+.. http://creativecommons.org/licenses/by/4.0
+
+========
+Abstract
+========
+
+In KVM4NFV project, we focus on the KVM hypervisor to enhance it for NFV,
+by looking at the following areas initially-
+
+* Minimal Interrupt latency variation for data plane VNFs:
+ * Minimal Timing Variation for Timing correctness of real-time VNFs
+ * Minimal packet latency variation for data-plane VNFs
+* Inter-VM communication
+* Fast live migration
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+
+.. http://creativecommons.org/licenses/by/4.0
+
+================================
+Using common platform components
+================================
+
+This section outlines basic usage principals and methods for some of the
+commonly deployed components of supported OPNFV scenario's in Colorado.
+The subsections provide an outline of how these components are commonly
+used and how to address them in an OPNFV deployment.The components derive
+from autonomous upstream communities and where possible this guide will
+provide direction to the relevant documentation made available by those
+communities to better help you navigate the OPNFV deployment.
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+
+.. http://creativecommons.org/licenses/by/4.0
+
+==========================
+Using Colorado Features
+==========================
+
+The following sections of the user guide provide feature specific usage
+guidelines and references for KVM4NFV CICD project.
+
+* <project>/docs/userguide/low_latency.userguide.rst
+* <project>/docs/userguide/live_migration.userguide.rst
+* <project>/docs/userguide/tuning.userguide.rst
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+
+.. http://creativecommons.org/licenses/by/4.0
+
+****************
+OPNFV User Guide
+****************
+Colorado 1.0
+------------
+
+.. toctree::
+ :maxdepth: 2
+
+ ./abstract.rst
+ ./introduction.rst
+ ./common.platform.render.rst
+ ./feature.userguide.render.rst
+ ./low_latency.userguide.rst
+ ./live_migration.userguide.rst
+ ./tuning.userguide.rst
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+
+.. http://creativecommons.org/licenses/by/4.0
+
+========
+Overview
+========
+
+The project "NFV Hypervisors-KVM" makes collaborative efforts to enable NFV
+features for existing hypervisors, which are not necessarily designed or
+targeted to meet the requirements for the NFVI.The KVM4NFV CICD scenario
+consists of Continuous Integration builds, deployments and testing
+combinations of virtual infrastructure components.
+
+KVM4NFV Features
+================
+
+Using this project, the following areas are targeted-
+
+* Minimal Interrupt latency variation for data plane VNFs:
+ * Minimal Timing Variation for Timing correctness of real-time VNFs
+ * Minimal packet latency variation for data-plane VNFs
+* Inter-VM communication
+* Fast live migration
+
+Some of the above items would require software development and/or specific
+hardware features, and some need just configurations information for the
+system (hardware, BIOS, OS, etc.).
+
+We include a requirements gathering stage as a formal part of the project.
+For each subproject, we will start with an organized requirement stage so
+that we can determine specific use cases (e.g. what kind of VMs should be
+live migrated) and requirements (e.g. interrupt latency, jitters, Mpps,
+migration-time, down-time, etc.) to set out the performance goals.
+
+Potential future projects would include:
+
+* Dynamic scaling (via scale-out) using VM instantiation
+* Fast live migration for SR-IOV
+
+The user guide outlines how to work with key components and features in
+the platform, each feature description section will indicate the scenarios
+that provide the components and configurations required to use it.
+
+The configuration guide details which scenarios are best for you and how to
+install and configure them.
+
+General usage guidelines
+========================
+
+The user guide for KVM4NFV CICD features and capabilities provide step by step
+instructions for using features that have been configured according to the
+installation and configuration instructions.
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+
.. http://creativecommons.org/licenses/by/4.0
-.. (c) <optionally add copywriters name>
Fast Live Migration
===================
-The NFV project requires fast live migration. The specific requirement is total
-live migration time < 2Sec, while keeping the VM down time < 10ms when running
-DPDK L2 forwarding workload.
+The NFV project requires fast live migration. The specific requirement is
+total live migration time < 2Sec, while keeping the VM down time < 10ms when
+running DPDK L2 forwarding workload.
We measured the baseline data of migrating an idle 8GiB guest running a DPDK L2
forwarding work load and observed that the total live migration time was 2271ms
is completed the VM down time is reduced to about 5-7ms.
b. Optimize zero page checking
Currently QEMU uses the SSE2 instruction to optimize the zero pages
- checking. The SSE2 instruction can process 16 bytes per instruction. By using
- the AVX2 instruction, we can process 32 bytes per instruction. Testingt shows
- that using AVX2 can speed up the zero pages checking process by about 25%.
+ checking. The SSE2 instruction can process 16 bytes per instruction.
+ By using the AVX2 instruction, we can process 32 bytes per instruction.
+ Testing shows that using AVX2 can speed up the zero pages checking process
+ by about 25%.
c. Remove unnecessary context synchronization.
The CPU context was being synchronized twice during live migration. Removing
this unnecessary synchronization shortened the VM downtime by about 100us.
Kernel: 4.2
QEMU v2.4.0
-Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
+Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit
+X540-AT2 (rev 01)
QEMU parameters:
::
- /root/qemu.git/x86_64-softmmu/qemu-system-x86_64-enable-kvm -cpu host -smp 4 –device virtio-net-pci,netdev=net1,mac=52:54:00:12:34:56 –netdev type=tap,id=net1,script=/etc/kvm/qemu-ifup,downscript=no,vhost=on–device virtio-net-pci,netdev=net2,mac=54:54:00:12:34:56 –netdevtype=tap,id=net2,script=/etc/kvm/qemu-ifup2,downscript=no,vhost=on -balloon virtio -m 8192-monitor stdio /mnt/liang/ia32e_rhel6u5.qcow
+${qemu} -smp ${guest_cpus} -monitor unix:${qmp_sock},server,nowait -daemonize \
+-cpu host,migratable=off,+invtsc,+tsc-deadline,pmu=off \
+-realtime mlock=on -mem-prealloc -enable-kvm -m 1G \
+-mem-path /mnt/hugetlbfs-1g \
+-drive file=/root/minimal-centos1.qcow2,cache=none,aio=threads \
+-netdev user,id=guest0,hostfwd=tcp:5555-:22 \
+-device virtio-net-pci,netdev=guest0 \
+-nographic -serial /dev/null -parallel /dev/null
Network connection
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+
+.. http://creativecommons.org/licenses/by/4.0
+
+Low Latency Environment
+=======================
+
+Achieving low latency with the KVM4NFV project requires setting up a special
+test environment. This environment includes the BIOS settings, kernel
+configuration, kernel parameters and the run-time environment.
+
+Hardware Environment Description
+--------------------------------
+
+BIOS setup plays an important role in achieving real-time latency. A collection
+of relevant settings, used on the platform where the baseline performance data
+was collected, is detailed below:
+
+CPU Features
+~~~~~~~~~~~~
+
+Some special CPU features like TSC-deadline timer, invariant TSC and Process
+posted interrupts, etc, are helpful for latency reduction.
+
+CPU Topology
+~~~~~~~~~~~~
+
+NUMA topology is also important for latency reduction.
+
+BIOS Setup
+~~~~~~~~~~
+
+Careful BIOS setup is important in achieving real time latency. Different
+platforms have different BIOS setups, below are the important BIOS settings on
+the platform used to collect the baseline performance data.
+
+Software Environment Setup
+--------------------------
+Both the host and the guest environment need to be configured properly to
+reduce latency variations. Below are some suggested kernel configurations.
+The ci/envs/ directory gives detailed implementation on how to setup the
+environment.
+
+Kernel Parameter
+~~~~~~~~~~~~~~~~
+
+Please check the default kernel configuration in the source code at:
+kernel/arch/x86/configs/opnfv.config.
+
+Below is host kernel boot line example:
+::
+isolcpus=11-15,31-35 nohz_full=11-15,31-35 rcu_nocbs=11-15,31-35
+iommu=pt intel_iommu=on default_hugepagesz=1G hugepagesz=1G mce=off idle=poll
+intel_pstate=disable processor.max_cstate=1 pcie_asmp=off tsc=reliable
+
+Below is guest kernel boot line example
+::
+isolcpus=1 nohz_full=1 rcu_nocbs=1 mce=off idle=poll default_hugepagesz=1G
+hugepagesz=1G
+
+Please refer to `tunning.userguide` for more explanation.
+
+Run-time Environment Setup
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Not only are special kernel parameters needed but a special run-time
+environment is also required. Please refer to `tunning.userguide` for
+more explanation.
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+
+.. http://creativecommons.org/licenses/by/4.0
+
+--------------------------------
+Colorado OpenStack User Guide
+--------------------------------
+
+OpenStack is a cloud operating system developed and released by the
+`OpenStack project <https://www.openstack.org>`_. OpenStack is used in OPNFV
+for controlling pools of compute, storage, and networking resources in a Pharos
+compliant infrastructure.
+
+OpenStack is used in Colorado to manage tenants (known in OpenStack as
+projects),users, services, images, flavours, and quotas across the Pharos
+infrastructure.The OpenStack interface provides the primary interface for an
+operational Colorado deployment and it is from the "horizon console" that an
+OPNFV user will perform the majority of administrative and operational
+activities on the deployment.
+
+OpenStack references
+--------------------
+
+The `OpenStack user guide <http://docs.openstack.org/user-guide>`_ provides
+details and descriptions of how to configure and interact with the OpenStack
+deployment.This guide can be used by lab engineers and operators to tune the
+OpenStack deployment to your liking.
+
+Once you have configured OpenStack to your purposes, or the Colorado
+deployment meets your needs as deployed, an operator, or administrator, will
+find the best guidance for working with OpenStack in the
+`OpenStack administration guide <http://docs.openstack.org/user-guide-admin>`_.
+
+Connecting to the OpenStack instance
+------------------------------------
+
+Once familiar with the basic of working with OpenStack you will want to connect
+to the OpenStack instance via the Horizon Console. The Horizon console provide
+a Web based GUI that will allow you operate the deployment.
+To do this you should open a browser on the JumpHost to the following address
+and enter the username and password:
+
+
+ http://{Controller-VIP}:80/index.html>
+ username: admin
+ password: admin
+
+Other methods of interacting with and configuring OpenStack,, like the REST API
+and CLI are also available in the Colorado deployment, see the
+`OpenStack administration guide <http://docs.openstack.org/user-guide-admin>`_
+for more information on using those interfaces.
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+
.. http://creativecommons.org/licenses/by/4.0
-.. (c) <optionally add copywriters name>
Low Latency Tunning Suggestion
==============================
-The correct configuration is critical for improving the NFV performance/latency.
-Even working on the same codebase, configurations can cause wildly different
-performance/latency results.
+The correct configuration is critical for improving the NFV
+performance/latency.Even working on the same codebase, configurations can cause
+wildly different performance/latency results.
There are many combinations of configurations, from hardware configuration to
Operating System configuration and application level configuration. And there
* **Power management:**
Most power management related features save power at the
expensive of latency. These features include: Intel®Turbo Boost Technology,
- Enhanced Intel®SpeedStep, Processor C state and P state. Normally they should
- be disabled but, depending on the real-time application design and latency
- requirements, there might be some features that can be enabled if the impact on
- deterministic execution of the workload is small.
+ Enhanced Intel®SpeedStep, Processor C state and P state. Normally they
+ should be disabled but, depending on the real-time application design and
+ latency requirements, there might be some features that can be enabled if
+ the impact on deterministic execution of the workload is small.
* **Hyper-Threading:**
The logic cores that share resource with other logic cores can introduce
* **SMI (System Management Interrupt):**
SMI runs outside of the kernel code and can potentially cause
latency. It is a pity there is no simple way to disable it. Some vendors may
- provide related switches in BIOS but most machines do not have this capability.
+ provide related switches in BIOS but most machines do not have this
+ capability.
Operating System Configuration
------------------------------
for more information.
* **Memory allocation:**
- Memory shoud be reserved for realtime applications and usually hugepage should
- be used to reduce page fauts/TLB misses.
+ Memory shoud be reserved for realtime applications and usually hugepage
+ should be used to reduce page fauts/TLB misses.
* **IRQ affinity:**
All the non-realtime IRQs should be affinitized to non realtime CPUs to
- reduce the impact on realtime CPUs. Some OS distributions contain an irqbalance
- daemon which balances the IRQs among all the cores dynamically. It should be
- disabled as well.
+ reduce the impact on realtime CPUs. Some OS distributions contain an
+ irqbalance daemon which balances the IRQs among all the cores dynamically.
+ It should be disabled as well.
* **Device assignment for VM:**
- If a device is used in a VM, then device passthrough is desirable. In this case,
- the IOMMU should be enabled.
+ If a device is used in a VM, then device passthrough is desirable. In this
+ case,the IOMMU should be enabled.
* **Tickless:**
- Frequent clock ticks cause latency. CONFIG_NOHZ_FULL should be enabled in the
- linux kernel. With CONFIG_NOHZ_FULL, the physical CPU will trigger many fewer
- clock tick interrupts(currently, 1 tick per second). This can reduce latency
- because each host timer interrupt triggers a VM exit from guest to host which
- causes performance/latency impacts.
+ Frequent clock ticks cause latency. CONFIG_NOHZ_FULL should be enabled in
+ the linux kernel. With CONFIG_NOHZ_FULL, the physical CPU will trigger many
+ fewer clock tick interrupts(currently, 1 tick per second). This can reduce
+ latency because each host timer interrupt triggers a VM exit from guest to
+ host which causes performance/latency impacts.
* **TSC:**
Mark TSC clock source as reliable. A TSC clock source that seems to be
- unreliable causes the kernel to continuously enable the clock source watchdog
- to check if TSC frequency is still correct. On recent Intel platforms with
- Constant TSC/Invariant TSC/Synchronized TSC, the TSC is reliable so the
- watchdog is useless but cause latency.
+ unreliable causes the kernel to continuously enable the clock source
+ watchdog to check if TSC frequency is still correct. On recent Intel
+ platforms with Constant TSC/Invariant TSC/Synchronized TSC, the TSC is
+ reliable so the watchdog is useless but cause latency.
* **Idle:**
The poll option forces a polling idle loop that can slightly improve the
* **Disable the RT throttling:**
RT Throttling is a Linux kernel mechanism that
- occurs when a process or thread uses 100% of the core, leaving no resources for
- the Linux scheduler to execute the kernel/housekeeping tasks. RT Throttling
- increases the latency so should be disabled.
+ occurs when a process or thread uses 100% of the core, leaving no resources
+ for the Linux scheduler to execute the kernel/housekeeping tasks. RT
+ Throttling increases the latency so should be disabled.
* **NUMA configuration:**
To achieve the best latency. CPU/Memory and device allocated for realtime
Code Review / kvmfornfv.git / commitdiff
