-==========================================================================
-OPNFV Release Notes for the Colorado release of OPNFV Apex deployment tool
-==========================================================================
+==============================================================================
+OPNFV Release Notes for the Colorado 3.0 release of OPNFV Apex deployment tool
+==============================================================================
.. contents:: Table of Contents
Abstract
========
-This document provides the release notes for Colorado release with the Apex
+This document provides the release notes for Colorado 3.0 release with the Apex
deployment toolchain.
License
( http://www.apache.org/licenses/ )
-Version history
+Version History
===============
| **Date** | **Ver.** | **Authors** | **Comment** |
| | | | |
+-------------+-----------+-----------------+----------------------+
+| 2016-12-05 | 2.3.0 | Tim Rozet | More updates for |
+| | | | Colorado 3.0 |
++-------------+-----------+-----------------+----------------------+
+| 2016-10-26 | 2.2.0 | Tim Rozet | More updates for |
+| | | | Colorado 2.0 |
++-------------+-----------+-----------------+----------------------+
+| 2016-09-20 | 2.1.0 | Tim Rozet | More updates for |
+| | | | Colorado |
++-------------+-----------+-----------------+----------------------+
| 2016-08-11 | 2.0.0 | Dan Radez | Updates for Colorado |
+-------------+-----------+-----------------+----------------------+
| 2015-09-17 | 1.0.0 | Dan Radez | Rewritten for |
| | | | RDO Manager update |
+-------------+-----------+-----------------+----------------------+
-Important notes
+Important Notes
===============
-This is the OPNFV Colorado release that implements the deploy stage of the
+This is the OPNFV Colorado 3.0 release that implements the deploy stage of the
OPNFV CI pipeline via Apex.
Apex is based on RDO's Triple-O installation tool chain.
Colorado release with the Apex deployment toolchain will establish an OPNFV
target system on a Pharos compliant lab infrastructure. The current definition
-of an OPNFV target system is and OpenStack Liberty combined with OpenDaylight
-Beryllium. The system is deployed with OpenStack High Availability (HA) for
-most OpenStack services. OpenDaylight is deployed in non-HA form as HA support
-is not availble for OpenDaylight at the time of the Colorado release. Ceph
+of an OPNFV target system is OpenStack Mitaka combined with an SDN
+controller, such as OpenDaylight. The system is deployed with OpenStack High
+Availability (HA) for most OpenStack services. SDN controllers are deployed
+only on the first controller (see HAIssues_ for known HA SDN issues). Ceph
storage is used as Cinder backend, and is the only supported storage for
Colorado. Ceph is setup as 3 OSDs and 3 Monitors, one OSD+Mon per Controller
-node.
+node in an HA setup. Apex also supports non-HA deployments, which deploys a
+single controller and n number of compute nodes. Furthermore, Apex is
+capable of deploying scenarios in a bare metal or virtual fashion. Virtual
+deployments use multiple VMs on the jump host and internal networking to
+simulate the a bare metal deployment.
- Documentation is built by Jenkins
- .iso image is built by Jenkins
- .rpm packages are built by Jenkins
- Jenkins deploys a Colorado release with the Apex deployment toolchain
- baremetal, which includes 3 control+network nodes, and 2 compute nodes.
+ bare metal, which includes 3 control+network nodes, and 2 compute nodes.
Release Data
============
| **Project** | apex |
| | |
+--------------------------------------+--------------------------------------+
-| **Repo/tag** | apex/colorado.1.0 |
+| **Repo/tag** | apex/colorado.3.0 |
| | |
+--------------------------------------+--------------------------------------+
-| **Release designation** | colorado.1.0 |
+| **Release designation** | colorado.3.0 |
| | |
+--------------------------------------+--------------------------------------+
-| **Release date** | 2016-09-14 |
+| **Release date** | 2016-12-5 |
| | |
+--------------------------------------+--------------------------------------+
-| **Purpose of the delivery** | OPNFV Colorado release |
+| **Purpose of the delivery** | OPNFV Colorado 3.0 release |
| | |
+--------------------------------------+--------------------------------------+
Module version changes
~~~~~~~~~~~~~~~~~~~~~~
-This is the first tracked version of the Colorado release with the Apex
+This is the second tracked version of the Colorado release with the Apex
deployment toolchain. It is based on following upstream versions:
- OpenStack (Mitaka release)
-- OpenDaylight (Beryllium release)
+- OpenDaylight (Beryllium/Boron releases)
- CentOS 7
-Document version changes
+Document Version Changes
~~~~~~~~~~~~~~~~~~~~~~~~
-This is the first tracked version of Colorado release with the Apex
-deployment toolchain.
The following documentation is provided with this release:
- OPNFV Installation instructions for the Colorado release with the Apex
- deployment toolchain - ver. 1.0.0
+ deployment toolchain - ver. 3.0
- OPNFV Release Notes for the Colorado release with the Apex deployment
- toolchain - ver. 1.0.0 (this document)
+ toolchain - ver. 2.3.0 (this document)
-Feature additions
+Feature Additions
~~~~~~~~~~~~~~~~~
+--------------------------------------+--------------------------------------+
| **JIRA REFERENCE** | **SLOGAN** |
| | |
+--------------------------------------+--------------------------------------+
-| JIRA: APEX-32 | Build.sh integration of RDO Manager |
-| | |
+| JIRA: APEX-107 | OpenDaylight HA - OVSDB Clustering |
+--------------------------------------+--------------------------------------+
-| JIRA: APEX-6 | Deploy.sh integration of RDO Manager |
-| | |
+| JIRA: APEX-108 | Migrate to OpenStack Mitaka |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-30 | Support VLAN tagged deployments |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-105 | Enable Huge Page Configuration |
+| | Options |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-111 | Allow RAM to be specified for |
+| | Control/Compute in Virtual |
+| | Deployments |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-119 | Enable OVS DPDK as a deployment |
+| | Scenario in Apex |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-126 | Tacker Service deployed by Apex |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-135 | Congress Service deployed by Apex |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-127 | Nova Instance CPU Pinning |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-130 | IPv6 Underlay Deployment |
+--------------------------------------+--------------------------------------+
-| JIRA: APEX-34 | Migrate and update Release |
-| | Documentation for Colorado |
+| JIRA: APEX-133 | FDIO with Honeycomb Agent |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-141 | Integrate VSPERF into Apex |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-172 | Enable ONOS SFC |
+--------------------------------------+--------------------------------------+
-Bug corrections
+Bug Corrections
~~~~~~~~~~~~~~~
**JIRA TICKETS:**
| **JIRA REFERENCE** | **SLOGAN** |
| | |
+--------------------------------------+--------------------------------------+
-| | |
-| | |
+| JIRA: APEX-86 | Need ability to specify number of |
+| | compute nodes |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-180 | Baremetal deployment error: Failed to|
+| | mount root partition /dev/sda on |
+| | /mnt/rootfs |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-161 | Heat autoscaling stack creation fails|
+| | for non-admin users |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-198 | Missing NAT iptables rule for public |
+| | network in instack VM |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-147 | Installer doesn't generate/distribute|
+| | SSH keys between compute nodes |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-109 | ONOS routes local subnet traffic to |
+| | GW |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-146 | Swift service present in available |
+| | endpoints |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-160 | Enable force_metadata to support |
+| | subnets with VM as the router |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-114 | OpenDaylight GUI is not available |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-100 | DNS1 and DNS2 should be handled in |
+| | nic bridging |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-100 | DNS1 and DNS2 should be handled in |
+| | nic bridging |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-155 | NIC Metric value not used when |
+| | bridging NICs |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-136 | 2 network deployment fails |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-89 | Deploy Ceph OSDs on compute nodes |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-137 | added arping ass dependency for |
+| | ONOS deployments |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-121 | VM Storage deletion intermittently |
+| | fails |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-182 | Nova services not correctly deployed |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-153 | brbm bridge not created in jumphost |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-274 | OVS DPDK scenario does not create |
+| | vhost user ports |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-277 | ONOS fails to launch due to OVS and |
+| | Kernel version mismatch |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-179 | DNS settings not working properly |
+| | for baremetal deployments |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-203 | Swift proxy enabled and fails in noha|
+| | deployment |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-230 | Tacker heat_uri setting is wrong |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-241 | Glance services are not running after|
+| | reboot |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-265 | Metadata problems with os-odl_l2-sfc |
+| | scenario |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-271 | L3 agent interface_driver set |
+| | incorrectly for nosdn fdio noha |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-275 | Metadata fails in ODL Boron |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-278 | Duplicate neutron config class |
+| | declaration for SDNVPN |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-281 | ONOS sometimes fails to provide |
+| | addresses to instances |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-283 | Change networking-vpp ML2 driver to |
+| | etcd |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-284 | vpp and hc versions for os-odl_l2 |
+| | -fdio-noha scenario |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-285 | Remove vlan-strip-offload off config |
+| | from os-odl_l2-fdio-noha scenario |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-286 | Make fdio repo an option in puppet- |
+| | fdio |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-289 | Set undercloud VM to autostart for |
+| | reboot |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-291 | link to apex installer missing on |
+| | OPNFV download site |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-293 | tenant_nic is assumed to be a virtual|
+| | name for nosdn fdio scenario |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-294 | dns server settings is ignored from |
+| | network_settings.yaml |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-297 | Add details in "11.2 Creating a Node |
+| | Inventory File" |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-326 | Consolidate SFC rpm into regular Apex|
+| | rpm |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-331 | Conditionally install OVS NSH in |
+| | ONOS Deployment |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-332 | IPv6 Scenario fails to deploy |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-333 | OVS DPDK does not work with UCS |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-334 | ONOS SFC not using SFC version of OVS|
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-335 | Performance settings missing in |
+| | os-odl_l2-fdio-ha.yaml |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-163 | Prefix to mask calculation doesn't |
+| | work |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-266 | ntp server shoudl be in network |
+| | settings |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-292 | unclear that ip ranges in network |
+| | settings may not overlap |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-349 | puppet-keystone failures in Colorado |
+| | errors deplyoment |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-344 | Update vpp/hc from 16.09 to 17.01 |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-346 | Update ODL to Carbon for FDIO |
+| | scenario |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-347 | Add deploy setting for ceph_device |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-341 | FDIO scenarios: not supplying |
+| | main-core and core-list workers |
+| | causes deploy failure |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-343 | Update qemu for fdio scenarios |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-344 | nosdn fdio scenario missing iommu |
+| | kernel params |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-337 | enable isolcpu on kernel and pin vpp |
+| | to the unused CPU for FDIO scenarios |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-338 | Install Procedure for baremetal |
+| | updated |
+--------------------------------------+--------------------------------------+
Deliverables
------------
-Software deliverables
+Software Deliverables
~~~~~~~~~~~~~~~~~~~~~
-Apex .iso file
-Apex overcloud .rpm (opnfv-apex)
-Apex undercloud .rpm (opnfv-apex-undercloud)
-Apex common .rpm (opnfv-apex-common)
-build.sh - Builds the above artifacts
-opnfv-deploy - Automatically deploys Target OPNFV System
-opnfv-clean - Automatically resets a Target OPNFV Deployment
-
-Documentation deliverables
+- Apex .iso file
+- Apex release .rpm (opnfv-apex-release)
+- Apex overcloud .rpm (opnfv-apex) - For nosdn and OpenDaylight Scenarios
+- Apex overcloud onos .rpm (opnfv-apex-onos) - ONOS Scenarios
+- Apex undercloud .rpm (opnfv-apex-undercloud)
+- Apex common .rpm (opnfv-apex-common)
+- build.sh - Builds the above artifacts
+- opnfv-deploy - Automatically deploys Target OPNFV System
+- opnfv-clean - Automatically resets a Target OPNFV Deployment
+- opnfv-util - Utility to connect to or debug Overcloud nodes + OpenDaylight
+
+Documentation Deliverables
~~~~~~~~~~~~~~~~~~~~~~~~~~
- OPNFV Installation instructions for the Colorado release with the Apex
- deployment toolchain - ver. 1.0.0
+ deployment toolchain - ver. 3.0
- OPNFV Release Notes for the Colorado release with the Apex deployment
- toolchain - ver. 1.0.0 (this document)
+ toolchain - ver. 2.2.0 (this document)
Known Limitations, Issues and Workarounds
=========================================
**Storage:** Ceph is the only supported storage configuration.
-**Min master requirements:** At least 16GB of RAM
+**Min master requirements:** At least 16GB of RAM for baremetal jumphost,
+24GB for virtual deployments (noHA).
-Known issues
+Known Issues
------------
**JIRA TICKETS:**
| **JIRA REFERENCE** | **SLOGAN** |
| | |
+--------------------------------------+--------------------------------------+
-| JIRA: APEX-89 | Deploy Ceph OSDs on the compute |
-| | nodes also |
+| JIRA: APEX-112 | ODL route local subnet traffic to GW |
+--------------------------------------+--------------------------------------+
-| JIRA: APEX-27 | OpenContrail Support |
-| | |
+| JIRA: APEX-149 | OpenFlow rules are populated slowly |
+--------------------------------------+--------------------------------------+
-| JIRA: APEX-30 | Support for VLAN tagged network |
-| | deployment architecture |
+| JIRA: APEX-268 | VMs with multiple floating IPs can |
+| | only access first NIC |
+--------------------------------------+--------------------------------------+
-| JIRA: APEX-100 | DNS1 and DNS2 not handled in |
-| | nic bridging |
+| JIRA: APEX-270 | Glance v2 api does not support url |
+| | images |
+--------------------------------------+--------------------------------------+
-| JIRA: APEX-47 | Integrate Tacker as part of SFC |
-| | Experimental Feature |
+| JIRA: APEX-272 | Enable VLAN tenant networking and |
+| | ability to configure range |
+--------------------------------------+--------------------------------------+
-| JIRA: APEX-84 | --flat option no longer working |
-| | |
+| JIRA: APEX-280 | Deleted network not cleaned up on |
+| | controller |
+--------------------------------------+--------------------------------------+
-| JIRA: APEX-51 | Integrate SDNVPN as a deploy option |
-| | |
+| JIRA: APEX-290 | Instances cannot connect to metadata |
+| | service after cluster reboot |
+--------------------------------------+--------------------------------------+
-| JIRA: APEX-99 | Syntax error when |
-| | running opnfv-deploy |
+| JIRA: APEX-295 | VLAN tenant networking does not work |
+| | in OVS DPDK scenario |
+--------------------------------------+--------------------------------------+
-| JIRA: APEX-86 | Compute node count configurable |
-| | for virtual deployments |
+| JIRA: APEX-339 | OVS DPDK PMD is not CPU pinned |
+--------------------------------------+--------------------------------------+
-| JIRA: APEX-141 | Adding VSPERF support |
-| | |
+| JIRA: APEX-355 | Undercloud CPU, RAM, Disk cannot be |
+| | adjusted |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-345 | Dashboard links time-out due to |
+| | running out of file descriptors for |
+| | mariadb |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-215 | Keystone services not configured and |
+| | the error is silently ignored (VLAN |
+| | Deployments) |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-208 | Need ability to specify which NIC to |
+| | place VLAN on |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-254 | Add dynamic hugepages configuration |
++--------------------------------------+--------------------------------------+
+| JIRA: APEX-138 | Unclear error message when interface |
+| | set to dhcp |
+--------------------------------------+--------------------------------------+
+
Workarounds
-----------
**-**
Scenario specific release notes
===============================
+Scenario os-odl_l3-nofeature known issues
+-----------------------------------------
+
+* `APEX-112 <https://jira.opnfv.org/browse/APEX-112>`_:
+ ODL routes local subnet traffic to GW
+
+Scenario os-odl_l2-nofeature known issues
+-----------------------------------------
+
+* `APEX-149 <https://jira.opnfv.org/browse/APEX-149>`_:
+ Openflow rules are populated very slowly
+
+Scenario os-odl_l2-bgpvpn known issues
+--------------------------------------
+
+None
+
+Scenario os-onos-nofeature known issues
+---------------------------------------
+
+None
+
+Scenario os-onos-sfc known issues
+---------------------------------
+
+* `APEX-281 <https://jira.opnfv.org/browse/APEX-281>`_:
+ ONOS sometimes fails to provide addresses to instances
+
+Scenario os-odl_l2-sfc-noha known issues
+----------------------------------------
+
+None
+
+Scenario os-nosdn-ovs known issues
+----------------------------------
+
+None
+
Scenario os-odl_l2-fdio-noha known issues
-----------------------------------------
-* `FDS-16 <https://jira.opnfv.org/browse/FDS-16>`_:
- Security group configuration through nova leads
- to vhostuser port connection issues
+* `FDS-121 <https://jira.opnfv.org/browse/FDS-121>`_:
+ qemu doesn't receive connection on socket, vhost-user reconnect problem
* `FDS-62 <https://jira.opnfv.org/browse/FDS-62>`_:
APEX - Increase number of files MariaDB can open
* `FDS-79 <https://jira.opnfv.org/browse/FDS-79>`_:
Sometimes (especially in bulk crete/delete operations
when multiple networks/ports are created within short time)
OpenDaylight doesn't accept creation requests
-* `FDS-80 <https://jira.opnfv.org/browse/FDS-80>`_:
- After launching a VM it stayed forever in BUILD status.
- Also further operation related to this VM (volume attachment etc.)
- caused problems
* `FDS-81 <https://jira.opnfv.org/browse/FDS-81>`_:
After functest finishes there are two bds on computes and
none on controller
-* `FDS-82 <https://jira.opnfv.org/browse/FDS-82>`_:
- Nova list shows no vms but there are some on computes in paused state
-
+* `APEX-217 <https://jira.opnfv.org/browse/APEX-217>`_:
+ qemu not configured with correct group:user
+* `APEX-337 <https://jira.opnfv.org/browse/APEX-337>`_:
+ enable isolcpu on kernel and pin vpp to unused cpu for fdio scenarios
+
+Scenario os-nosdn-fdio-noha known issues
+----------------------------------------
+
+Note that a set of manual configration steps need to be performed
+post an automated deployment for the scenario to be fully functional.
+Please refer to `FDS-159 <https://jira.opnfv.org/browse/FDS-159>`_ and
+`FDS-160 <https://jira.opnfv.org/browse/FDS-160>`_ for details.
+
+* `FDS-155 <https://jira.opnfv.org/browse/FDS-155>`_:
+ os-nosdn-fdio-noha scenario: tempest_smoke_serial causes
+ mariadb/mysqld process to hang
+* `FDS-156 <https://jira.opnfv.org/browse/FDS-156>`_:
+ os-nosdn-fdio-noha scenario: Race conditions for
+ network-vif-plugged notification
+* `FDS-157 <https://jira.opnfv.org/browse/FDS-157>`_:
+ os-nosdn-fdio-noha scenario: Intermittently VMs
+ would get assigned 2 IPs instead of 1
+* `FDS-158 <https://jira.opnfv.org/browse/FDS-158>`_:
+ os-nosdn-fdio-noha scenario: VM start/launch fails with
+ "no more IP addresses" in neutron logs
+* `FDS-159 <https://jira.opnfv.org/browse/FDS-159>`_:
+ os-nosdn-fdio-noha scenario: Security groups not yet supported
+* `FDS-160 <https://jira.opnfv.org/browse/FDS-160>`_:
+ os-nosdn-fdio-noha scenario: Vlan fix on controller
+* `FDS-161 <https://jira.opnfv.org/browse/FDS-161>`_:
+ os-nosdn-fdio-noha scenario: VPP fails with certain UCS B-series blades
+
+.. _HAIssues:
General HA scenario known issues
--------------------------------
* `COPPER-22 <https://jira.opnfv.org/browse/COPPER-22>`_:
Congress service HA deployment is not yet supported/verified.
+* `APEX-276 <https://jira.opnfv.org/browse/APEX-276>`_:
+ ODL HA unstable and crashes frequently
Test Result
===========
The Colorado release with the Apex deployment toolchain has undergone QA
-test runs with the following results:
-
-+--------------------------------------+--------------------------------------+
-| **TEST-SUITE** | **Results:** |
-| | |
-+--------------------------------------+--------------------------------------+
-| **-** | **-** |
-+--------------------------------------+--------------------------------------+
+test runs with the OPNFV FuncTest project. The latest results per scenario
+can be found `here <http://testresults.opnfv.org/reporting/functest/
+release/colorado/index-status-apex.html>`_ for Colorado release.
References
:Authors: Tim Rozet (trozet@redhat.com)
:Authors: Dan Radez (dradez@redhat.com)
-:Version: 1.0.0
+:Version: 2.3.0
Code Review / apex.git / blobdiff