+++ /dev/null
-General Requirements Background and Terminology\r
------------------------------------------------\r
-\r
-Terminologies and definitions\r
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r
-\r
-NFVI\r
- The term is an abbreviation for Network Function Virtualization\r
- Infrastructure; sometimes it is also referred as data plane in this\r
- document. The NFVI provides the virtual resources to the virtual\r
- network functions under the control of the VIM.\r
-\r
-VIM\r
- The term is an abbreviation for Virtual Infrastructure Manager;\r
- sometimes it is also referred as control plane in this document.\r
- The VIM controls and manages the NFVI compute, network and storage\r
- resources to provide the required virtual resources to the VNFs.\r
-\r
-Operator\r
- The term refers to network service providers and Virtual Network\r
- Function (VNF) providers.\r
-\r
-End-User\r
- The term refers to a subscriber of the Operator's services.\r
-\r
-Network Service\r
- The term refers to a service provided by an Operator to its\r
- end-users using a set of (virtualized) Network Functions\r
-\r
-Infrastructure Services\r
- The term refers to services provided by the NFV Infrastructure to the VNFs\r
- as required by the Management & Orchestration functions and especially the VIM.\r
- I.e. these are the virtual resources as perceived by the VNFs.\r
-\r
-Smooth Upgrade\r
- The term refers to an upgrade that results in no service outage\r
- for the end-users.\r
-\r
-Rolling Upgrade\r
- The term refers to an upgrade strategy, which upgrades a node or a subset\r
- of nodes at a time in a wave style rolling through the data centre. It\r
- is a popular upgrade strategy to maintain service availability.\r
-\r
-Parallel Universe Upgrade\r
- The term refers to an upgrade strategy, which creates and deploys\r
- a new universe - a system with the new configuration - while the old\r
- system continues running. The state of the old system is transferred\r
- to the new system after sufficient testing of the new system.\r
-\r
-Infrastructure Resource Model\r
- The term refers to the representation of infrastructure resources,\r
- namely: the physical resources, the virtualization\r
- facility resources and the virtual resources.\r
-\r
-Physical Resource\r
- The term refers to a piece of hardware in the NFV infrastructure that may\r
- also include firmware enabling this piece of hardware.\r
-\r
-Virtual Resource\r
- The term refers to a resource, which is provided as services built on top\r
- of the physical resources via the virtualization facilities; in particular,\r
- virtual resources are the resources on which VNFs are deployed. Examples of\r
- virtual resources are: VMs, virtual switches, virtual routers, virtual disks.\r
-\r
-Visualization Facility\r
- The term refers to a resource that enables the creation\r
- of virtual environments on top of the physical resources, e.g.\r
- hypervisor, OpenStack, etc.\r
-\r
-Upgrade Campaign\r
- The term refers to a choreography that describes how the upgrade should\r
- be performed in terms of its targets (i.e. upgrade objects), the\r
- steps/actions required of upgrading each, and the coordination of these\r
- steps so that service availability can be maintained. It is an input to an\r
- upgrade tool (Escalator) to carry out the upgrade.\r
-\r
-Upgrade Duration\r
- The duration of an upgrade characterized by the time elapsed between its\r
- initiation and its completion. E.g. from the moment the execution of an\r
- upgrade campaign has started until it has been committed. Depending on\r
- the upgrade strategy, the state of the configuration and the upgrade target\r
- some parts of the system may be in a more vulnerable state with respect to\r
- service availbility.\r
-\r
-Outage\r
- The period of time during which a given service is not provided is referred\r
- as the outage of that given service. If a subsystem or the entire system\r
- does not provide any service, it is the outage of the given subsystem or the\r
- system. Smooth upgrade means upgrade with no outage for the user plane, i.e.\r
- no VNF should experience service outage.\r
-\r
-Rollback\r
- The term refers to a failure handling strategy that reverts the changes\r
- done by a potentially failed upgrade execution one by one in a reverse order.\r
- I.e. it is like undoing the changes done by the upgrade.\r
-\r
-Backup\r
- The term refers to data persisted to a storage, so that it can be used to\r
- restore the system or a given part of it in the same state as it was when the\r
- backup was created assuming a cold restart. Changes made to the system from\r
- the moment the backup was created till the moment it is used to restore the\r
- (sub)system are lost in the restoration process.\r
-\r
-Restore\r
- The term refers to a failure handling strategy that reverts the changes\r
- done, for example, by an upgrade by restoring the system from some backup\r
- data. This results in the loss of any change and data persisted after the\r
- backup was been taken. To recover those additional measures need to be taken\r
- if necessary (e.g. rollforward).\r
-\r
-Rollforward\r
- The term refers to a failure handling strategy applied after a restore\r
- (from a backup) opertaion to recover any loss of data persisted between\r
- the time the backup has been taken and the moment it is restored. Rollforward\r
- requires that data that needs to survive the restore operation is logged at\r
- a location not impacted by the restore so that it can be re-applied to the\r
- system after its restoration from the backup.\r
-\r
-Downgrade\r
- The term refers to an upgrade in which an earlier version of the software\r
- is restored through the upgrade procedure. A system can be downgraded to any\r
- earlier version and the compatibility of the versions will determine the\r
- applicable upgrade strategies and whether service outage can be avoided.\r
- In particular any data conversion needs special attention.\r
-\r
-\r
-\r
-Upgrade Objects\r
-~~~~~~~~~~~~~~~\r
-\r
-Physical Resource\r
-^^^^^^^^^^^^^^^^^\r
-\r
-Most cloud infrastructures support the dynamic addition and removal of\r
-hardware. Accordingly a hardware upgrade could be done by adding the new\r
-piece of hardware and removing the old one. From the persepctive of smooth\r
-upgrade the orchestration/scheduling of these actions is the primary concern.\r
-\r
-Upgrading a physical resource may involve as well the upgrade of its firmware\r
-and/or modifying its configuration data. This may require the restart of the\r
-hardware.\r
-\r
-\r
-\r
-Virtual Resources\r
-^^^^^^^^^^^^^^^^^\r
-\r
-Addition and removal of virtual resources may be initiated by the users or be\r
-a result of an elasticity action. Users may also request the upgrade of their\r
-virtual resources using a new VM image.\r
-\r
-.. Needs to be moved to requirement section: Escalator should facilitate such an\r
-option and allow for a smooth upgrade.\r
-\r
-On the other hand changes in the infrastructure, namely, in the hardware and/or\r
-the virtualization facility resources may result in the upgrade of the virtual\r
-resources. For example if by some reason the hypervisor is changed and\r
-the current VMs cannot be migrated to the new hypervisor - they are\r
-incompatible - then the VMs need to be upgraded too. This is not\r
-something the NFVI user (i.e. VNFs ) would know about. \r
-\r
-\r
-Virtualization Facility Resources\r
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r
-\r
-Based on the functionality they provide, virtualization facility\r
-resources could be divided into computing node, networking node,\r
-storage node and management node.\r
-\r
-The possible upgrade objects in these nodes are considered below:\r
-(Note: hardware based virtualization may be considered as virtualization\r
-facility resource, but from escalator perspective, it is better to\r
-consider it as part of the hardware upgrade. )\r
-\r
-**Computing node**\r
-\r
-1. OS Kernel\r
-\r
-2. Hypvervisor and virtual switch\r
-\r
-3. Other kernel modules, like drivers\r
-\r
-4. User space software packages, like nova-compute agents and other\r
- control plane programs.\r
-\r
-Updating 1 and 2 will cause the loss of virtualzation functionality of\r
-the compute node, which may lead to the interruption of data plane services \r
-if the virtual resource is not redudant.\r
-\r
-Updating 3 might have the same result.\r
-\r
-Updating 4 might lead to control plane services interruption if not an\r
-HA deployment.\r
-\r
-.. <MT> I'm not sure why would 4 cause control plane interruption on a\r
- compute node. My understanding is that simply the node cannot be managed.\r
- Redundancy won't help in that either.\r
-\r
-\r
-**Networking node**\r
-\r
-1. OS kernel, optional, not all switches/routers allow the upgrade their\r
- OS since it is more like a firmware than a generic OS.\r
-\r
-2. User space software package, like neutron agents and other control\r
- plane programs\r
-\r
-Updating 1 if allowed will cause a node reboot and therefore leads to\r
-data plane service interruption if the virtual resource is not\r
-redundant.\r
-\r
-Updating 2 might lead to control plane services interruption if not an\r
-HA deployment.\r
-\r
-**Storage node**\r
-\r
-1. OS kernel, optional, not all storage nodes allow the upgrade their OS\r
- since it is more like a firmware than a generic OS.\r
-\r
-2. Kernel modules\r
-\r
-3. User space software packages, control plane programs\r
-\r
-Updating 1 if allowed will cause a node reboot and therefore leads to\r
-data plane services interruption if the virtual resource is not\r
-redundant.\r
-\r
-Update 2 might result in the same.\r
-\r
-Updating 3 might lead to control plane services interruption if not an\r
-HA deployment.\r
-\r
-**Management node**\r
-\r
-1. OS Kernel\r
-\r
-2. Kernel modules, like driver\r
-\r
-3. User space software packages, like database, message queue and\r
- control plane programs.\r
-\r
-Updating 1 will cause a node reboot and therefore leads to control\r
-plane services interruption if not an HA deployment. Updating 2 might\r
-result in the same.\r
-\r
-Updating 3 might lead to control plane services interruption if not an\r
-HA deployment.\r
-\r
-\r
-\r
-\r
-\r
-Upgrade Granularity\r
-~~~~~~~~~~~~~~~~~~~\r
-\r
-The granularity of an upgrade can be characterized from two perspective:\r
-- the physical dimension and\r
-- the software dimension\r
-\r
-\r
-Physical Dimension\r
-^^^^^^^^^^^^^^^^^^\r
-\r
-The physical dimension characterizes the number of similar upgrade objects\r
-targeted by the upgrade, i.e. whether it is full / partial upgrade of a\r
-data centre, cluster, zone.\r
-Because of the upgrade of a data centre or a zone, it may be divided into\r
-several batches. Thus there is a need for efficiency in the execution of\r
-upgrades of potentially huge number of upgrade objects while still maintain\r
-availability to fulfill the requirement of smooth upgrade.\r
-\r
-The upgrade of a cloud environment (cluster) may also\r
-be partial. For example, in one cloud environment running a number of\r
-VNFs, we may just try to upgrade one of them to check the stability and\r
-performance, before we upgrade all of them.\r
-Thus there is a need for proper organization of the artifacts associated with\r
-the different upgrade objects. Also the different versions should be able\r
-to coextist beyond the upgrade period.\r
-\r
-From this perspective special attention may be needed when upgrading\r
-objects that are collaborating in a redundancy schema as in this case\r
-different versions not only need to coexist but also collaborate. This\r
-puts requirement on the upgrade objects primarily. If this is not possible\r
-the upgrade campaign should be designed in such a way that the proper\r
-isolation is ensured.\r
-\r
-Software Dimension\r
-^^^^^^^^^^^^^^^^^^\r
-\r
-The software dimension of the upgrade characterizes the upgrade object\r
-type targeted and the combination in which they are upgraded together.\r
-\r
-Even though the upgrade may\r
-initially target only one type of upgrade object, e.g. the hypervisor\r
-the dependency of other upgrade objects on this initial target object may\r
-require their upgrade as well. I.e. the upgrades need to be combined. From this\r
-perspective the main concern is compatibility of the dependent and\r
-sponsor objects. To take into consideration of these dependencies\r
-they need to be described together with the version compatility information.\r
-Breaking dependencies is the major cause of outages during upgrades.\r
-\r
-In other cases it is more efficient to upgrade a combination of upgrade\r
-objects than to do it one by one. One aspect of the combination is how\r
-the upgrade packages can be combined, whether a new image can be created for\r
-them before hand or the different packages can be installed during the upgrade\r
-independently, but activated together.\r
-\r
-The combination of upgrade objects may span across\r
-layers (e.g. software stack in the host and the VM of the VNF).\r
-Thus, it may require additional coordination between the management layers.\r
-\r
-With respect to each upgrade object type and even stacks we can\r
-distingush major and minor upgrades:\r
-\r
-**Major Upgrade**\r
-\r
-Upgrades between major releases may introducing significant changes in\r
-function, configuration and data, such as the upgrade of OPNFV from\r
-Arno to Brahmaputra.\r
-\r
-**Minor Upgrade**\r
-\r
-Upgrades inside one major releases which would not leads to changing\r
-the structure of the platform and may not infect the schema of the\r
-system data.\r
-\r
-Scope of Impact\r
-~~~~~~~~~~~~~~~\r
-\r
-Considering availability and therefore smooth upgrade, one of the major\r
-concerns is the predictability and control of the outcome of the different\r
-upgrade operations. Ideally an upgrade can be performed without impacting any\r
-entity in the system, which means none of the operations change or potentially\r
-change the behaviour of any entity in the system in an uncotrolled manner.\r
-Accordingly the operations of such an upgrade can be performed any time while\r
-the system is running, while all the entities are online. No entity needs to be\r
-taken offline to avoid such adverse effects. Hence such upgrade operations\r
-are referred as online operations. The effects of the upgrade might be activated\r
-next time it is used, or may require a special activation action such as a\r
-restart. Note that the activation action provides more control and predictability.\r
-\r
-If an entity's behavior in the system may change due to the upgrade it may\r
-be better to take it offline for the time of the relevant upgrade operations.\r
-The main question is however considering the hosting relation of an upgrade\r
-object what hosted entities are impacted. Accordingly we can identify a scope\r
-which is impacted by taking the given upgrade object offline. The entities\r
-that are in the scope of impact may need to be taken offline or moved out of\r
-this scope i.e. migrated.\r
-\r
-If the impacted entity is in a different layer managed by another manager\r
-this may require coordination because taking out of service some\r
-infrastructure resources for the time of their upgrade which support virtual\r
-resources used by VNFs that should not experience outages. The hosted VNFs\r
-may or may not allow for the hot migration of their VMs. In case of migration\r
-the VMs placement policy should be considered.\r
-\r
-\r
-\r
-Upgrade duration\r
-~~~~~~~~~~~~~~~~\r
-\r
-As the OPNFV end-users are primarily Telecom operators, the network\r
-services provided by the VNFs deployed on the NFVI should meet the\r
-requirement of 'Carrier Grade'.::\r
-\r
- In telecommunication, a "carrier grade" or"carrier class" refers to a\r
- system, or a hardware or software component that is extremely reliable,\r
- well tested and proven in its capabilities. Carrier grade systems are\r
- tested and engineered to meet or exceed "five nines" high availability\r
- standards, and provide very fast fault recovery through redundancy\r
- (normally less than 50 milliseconds). [from wikipedia.org]\r
-\r
-"five nines" means working all the time in ONE YEAR except 5'15".\r
-\r
-::\r
-\r
- We have learnt that a well prepared upgrade of OpenStack needs 10\r
- minutes. The major time slot in the outage time is used spent on\r
- synchronizing the database. [from ' Ten minutes OpenStack Upgrade? Done!\r
- ' by Symantec]\r
-\r
-This 10 minutes of downtime of the OpenStack services however did not impact the\r
-users, i.e. the VMs running on the compute nodes. This was the outage of\r
-the control plane only. On the other hand with respect to the\r
-preparations this was a manually tailored upgrade specific to the\r
-particular deployment and the versions of each OpenStack service.\r
-\r
-The project targets to achieve a more generic methodology, which however\r
-requires that the upgrade objects fulfil certain requirements. Since\r
-this is only possible on the long run we target first the upgrade\r
-of the different VIM services from version to version.\r
-\r
-**Questions:**\r
-\r
-1. Can we manage to upgrade OPNFV in only 5 minutes?\r
- \r
-.. <MT> The first question is whether we have the same carrier grade\r
- requirement on the control plane as on the user plane. I.e. how\r
- much control plane outage we can/willing to tolerate?\r
- In the above case probably if the database is only half of the size\r
- we can do the upgrade in 5 minutes, but is that good? It also means\r
- that if the database is twice as much then the outage is 20\r
- minutes.\r
- For the user plane we should go for less as with two release yearly\r
- that means 10 minutes outage per year.\r
-\r
-.. <Malla> 10 minutes outage per year to the users? Plus, if we take\r
- control plane into the consideration, then total outage will be\r
- more than 10 minute in whole network, right?\r
-\r
-.. <MT> The control plane outage does not have to cause outage to\r
- the users, but it may of course depending on the size of the system\r
- as it's more likely that there's a failure that needs to be handled\r
- by the control plane.\r
-\r
-2. Is it acceptable for end users ? Such as a planed service\r
- interruption will lasting more than ten minutes for software\r
- upgrade.\r
-\r
-.. <MT> For user plane, no it's not acceptable in case of\r
- carrier-grade. The 5' 15" downtime should include unplanned and\r
- planned downtimes.\r
- \r
-.. <Malla> I go agree with Maria, it is not acceptable.\r
-\r
-3. Will any VNFs still working well when VIM is down?\r
-\r
-.. <MT> In case of OpenStack it seems yes. .:)\r
-\r
-The maximum duration of an upgrade\r
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r
-\r
-The duration of an upgrade is related to and proportional with the\r
-scale and the complexity of the OPNFV platform as well as the\r
-granularity (in function and in space) of the upgrade.\r
-\r
-.. <Malla> Also, if is a partial upgrade like module upgrade, it depends\r
- also on the OPNFV modules and their tight connection entities as well.\r
-\r
-.. <MT> Since the maintenance window is shrinking and becoming non-existent\r
- the duration of the upgrade is secondary to the requirement of smooth upgrade.\r
- But probably we want to be able to put a time constraint on each upgrade\r
- during which it must complete otherwise it is considered failed and the system\r
- should be rolled back. I.e. in case of automatic execution it might not be clear\r
- if an upgrade is long or just hanging. The time constraints may be a function\r
- of the size of the system in terms of the upgrade object(s).\r
-\r
-The maximum duration of a roll back when an upgrade is failed \r
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r
-\r
-The duration of a roll back is short than the corresponding upgrade. It\r
-depends on the duration of restore the software and configure data from\r
-pre-upgrade backup / snapshot.\r
-\r
-.. <MT> During the upgrade process two types of failure may happen:\r
- In case we can recover from the failure by undoing the upgrade\r
- actions it is possible to roll back the already executed part of the\r
- upgrade in graceful manner introducing no more service outage than\r
- what was introduced during the upgrade. Such a graceful roll back\r
- requires typically the same amount of time as the executed portion of\r
- the upgrade and impose minimal state/data loss.\r
- \r
-.. <MT> Requirement: It should be possible to roll back gracefully the\r
- failed upgrade of stateful services of the control plane.\r
- In case we cannot recover from the failure by just undoing the\r
- upgrade actions, we have to restore the upgraded entities from their\r
- backed up state. In other terms the system falls back to an earlier\r
- state, which is typically a faster recovery procedure than graceful\r
- roll back and depending on the statefulness of the entities involved it\r
- may result in significant state/data loss.\r
- \r
-.. <MT> Two possible types of failures can happen during an upgrade\r
-\r
-.. <MT> We can recover from the failure that occurred in the upgrade process:\r
- In this case, a graceful rolling back of the executed part of the\r
- upgrade may be possible which would "undo" the executed part in a\r
- similar fashion. Thus, such a roll back introduces no more service\r
- outage during an upgrade than the executed part introduced. This\r
- process typically requires the same amount of time as the executed\r
- portion of the upgrade and impose minimal state/data loss.\r
-\r
-.. <MT> We cannot recover from the failure that occurred in the upgrade\r
- process: In this case, the system needs to fall back to an earlier\r
- consistent state by reloading this backed-up state. This is typically\r
- a faster recovery procedure than the graceful roll back, but can cause\r
- state/data loss. The state/data loss usually depends on the\r
- statefulness of the entities whose state is restored from the backup.\r
-\r
-The maximum duration of a VNF interruption (Service outage)\r
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r
-\r
-Since not the entire process of a smooth upgrade will affect the VNFs,\r
-the duration of the VNF interruption may be shorter than the duration\r
-of the upgrade. In some cases, the VNF running without the control\r
-from of the VIM is acceptable.\r
-\r
-.. <MT> Should require explicitly that the NFVI should be able to\r
- provide its services to the VNFs independent of the control plane?\r
-\r
-.. <MT> Requirement: The upgrade of the control plane must not cause\r
- interruption of the NFVI services provided to the VNFs.\r
-\r
-.. <MT> With respect to carrier-grade the yearly service outage of the\r
- VNF should not exceed 5' 15" regardless whether it is planned or\r
- unplanned outage. Considering the HA requirements TL-9000 requires an\r
- end-to-end service recovery time of 15 seconds based on which the ETSI\r
- GS NFV-REL 001 V1.1.1 (2015-01) document defines three service\r
- availability levels (SAL). The proposed example service recovery times\r
- for these levels are:\r
-\r
-.. <MT> SAL1: 5-6 seconds\r
-\r
-.. <MT> SAL2: 10-15 seconds\r
-\r
-.. <MT> SAL3: 20-25 seconds\r
-\r
-.. <Pva> my comment was actually that the downtime metrics of the\r
- underlying elements, components and services are small fraction of the\r
- total E2E service availability time. No-one on the E2E service path\r
- will get the whole downtime allocation (in this context it includes\r
- upgrade process related outages for the services provided by VIM etc.\r
- elements that are subject to upgrade process).\r
- \r
-.. <MT> So what you are saying is that the upgrade of any entity\r
- (component, service) shouldn't cause even this much service\r
- interruption. This was the reason I brought these figures here as well\r
- that they are posing some kind of upper-upper boundary. Ideally the\r
- interruption is in the millisecond range i.e. no more than a\r
- switch-over or a live migration.\r
- \r
-.. <MT> Requirement: Any interruption caused to the VNF by the upgrade\r
- of the NFVI should be in the sub-second range.\r
-\r
-.. <MT]> In the future we also need to consider the upgrade of the NFVI,\r
- i.e. HW, firmware, hypervisors, host OS etc.
\ No newline at end of file
+++ /dev/null
-Functional Requirements
------------------------
-
-Basic Actions
-~~~~~~~~~~~~~
-
-This section describes the basic functions may required by Escalator.
-
-Preparation (offline)
-^^^^^^^^^^^^^^^^^^^^^
-
-This is the design phase when the upgrade plan (or upgrade campaign) is
-being designed so that it can be executed automatically with minimal
-service outage. It may include the following work:
-
-1. Check the dependencies of the software modules and their impact,
- backward compatibilities to figure out the appropriate upgrade method
- and ordering.
-2. Find out if a rolling upgrade could be planned with several rolling
- steps to avoid any service outage due to the upgrade some
- parts/services at the same time.
-3. Collect the proper version files and check the integration for
- upgrading.
-4. The preparation step should produce an output (i.e. upgrade
- campaign/plan), which is executable automatically in an NFV Framework
- and which can be validated before execution.
-
- - The upgrade campaign should not be referring to scalable entities
- directly, but allow for adaptation to the system configuration and
- state at any given moment.
- - The upgrade campaign should describe the ordering of the upgrade
- of different entities so that dependencies, redundancies can be
- maintained during the upgrade execution
- - The upgrade campaign should provide information about the
- applicable recovery procedures and their ordering.
- - The upgrade campaign should consider information about the
- verification/testing procedures to be performed during the upgrade
- so that upgrade failures can be detected as soon as possible and
- the appropriate recovery procedure can be identified and applied.
- - The upgrade campaign should provide information on the expected
- execution time so that hanging execution can be identified
- - The upgrade campaign should indicate any point in the upgrade when
- coordination with the users (VNFs) is required.
-
-.. <hujie> Depends on the attributes of the object being upgraded, the
- upgrade plan may be slitted into step(s) and/or sub-plan(s), and even
- more small sub-plans in design phase. The plan(s) or sub-plan(s) my
- include step(s) or sub-plan(s).
-
-Validation the upgrade plan / Checking the pre-requisites of System( offline / online)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The upgrade plan should be validated before the execution by testing
-it in a test environment which is similar to the product environment.
-
-.. <MT> However it could also mean that we can identify some properties
- that it should satisfy e.g. what operations can or cannot be executed
- simultaneously like never take out two VMs of the same VNF.
-
-.. <MT> Another question is if it requires that the system is in a particular
- state when the upgrade is applied. I.e. if there's certain amount of
- redundancy in the system, migration is enabled for VMs, when the NFVI
- is upgraded the VIM is healthy, when the VIM is upgraded the NFVI is
- healthy, etc.
-
-.. <MT> I'm not sure what online validation means: Is it the validation of the
- upgrade plan/campaign or the validation of the system that it is in a
- state that the upgrade can be performed without too much risk?==
-
-Before the upgrade plan being executed, the system healthy of the
-online product environment should be checked and confirmed to satisfy
-the requirements which were described in the upgrade plan. The
-sysinfo, e.g. which included system alarms, performance statistics and
-diagnostic logs, will be collected and analogized. It is required to
-resolve all of the system faults or exclude the unhealthy part before
-executing the upgrade plan.
-
-
-Backup/Snapshot (online)
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-For avoid loss of data when a unsuccessful upgrade was encountered, the
-data should be back-upped and the system state snapshot should be taken
-before the execution of upgrade plan. This would be considered in the
-upgrade plan.
-
-Several backups/Snapshots may be generated and stored before the single
-steps of changes. The following data/files are required to be
-considered:
-
-1. running version files for each node.
-2. system components' configuration file and database.
-3. image and storage, if it is necessary.
-
-.. <MT> Does 3 imply VNF image and storage? I.e. VNF state and data?==
-
-.. <hujie> The following text is derived from previous "4. Negotiate
- with the VNF if it's ready for the upgrade"
-
-Although the upper layer, which include VNFs and VNFMs, is out of the
-scope of Escalator, but it is still recommended to let it ready for a
-smooth system upgrade. The escalator could not guarantee the safe of
-VNFs. The upper layer should have some safe guard mechanism in design,
-and ready for avoiding failure in system upgrade.
-
-Execution (online)
-^^^^^^^^^^^^^^^^^^
-
-The execution of upgrade plan should be a dynamical procedure which is
- controlled by Escalator.
-
-.. <hujie> Revised text to be general.==
-
-1. It is required to supporting execution ether in sequence or in
- parallel.
-2. It is required to check the result of the execution and take the
- action according the situation and the policies in the upgrade plan.
-3. It is required to execute properly on various configurations of
- system object. I.e. stand-alone, HA, etc.
-4. It is required to execute on the designated different parts of the
- system. I.e. physical server, virtualized server, rack, chassis,
- cluster, even different geographical places.
-
-Testing (online)
-^^^^^^^^^^^^^^^^
-
-The testing after upgrade the whole system or parts of system to make
-sure the upgraded system(object) is working normally.
-
-.. <hujie> Revised text to be general.
-
-1. It is recommended to run the prepared test cases to see if the
- functionalities are available without any problem.
-2. It is recommended to check the sysinfo, e.g. system alarms,
- performance statistics and diagnostic logs to see if there are any
- abnormal.
-
-Restore/Roll-back (online)
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-When upgrade is failure unfortunately, a quick system restore or system
-roll-back should be taken to recovery the system and the services.
-
-.. <hujie> Revised text to be general.
-
-1. It is recommend to support system restore from backup when upgrade
- was failed.
-2. It is recommend to support graceful roll-back with reverse order
- steps if possible.
-
-Monitoring (online)
-^^^^^^^^^^^^^^^^^^^
-
-Escalator should continually monitor the process of upgrade. It is
-keeping update status of each module, each node, each cluster into a
-status table during upgrade.
-
-.. <hujie> Revised text to be general.
-
-1. It is required to collect the status of every objects being upgraded
- and sending abnormal alarms during the upgrade.
-2. It is recommend to reuse the existing monitoring system, like alarm.
-3. It is recommend to support pro-actively query.
-4. It is recommend to support passively wait for notification.
-
-**Two possible ways for monitoring:**
-
-**Pro-Actively Query** requires NFVI/VIM provides proper API or CLI
-interface. If Escalator serves as a service, it should pass on these
-interfaces.
-
-**Passively Wait for Notification** requires Escalator provides
-callback interface, which could be used by NFVI/VIM systems or upgrade
-agent to send back notification.
-
-.. <hujie> I am not sure why not to subscribe the notification.
-
-Logging (online)
-^^^^^^^^^^^^^^^^
-
-Record the information generated by escalator into log files. The log
-file is used for manual diagnostic of exceptions.
-
-1. It is required to support logging.
-2. It is recommended to include time stamp, object id, action name,
- error code, etc.
-
-Administrative Control (online)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Administrative Control is used for control the privilege to start any
-escalator's actions for avoiding unauthorized operations.
-
-#. It is required to support administrative control mechanism
-#. It is recommend to reuse the system's own secure system.
-#. It is required to avoid conflicts when the system's own secure system
- being upgraded.
-
-Requirements on Object being upgraded
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-.. <hujie> We can develop BPs in future from requirements of this section and
- gap analysis for upper stream projects
-
-Escalator focus on smooth upgrade. In practical implementation, it
-might be combined with installer/deplorer, or act as an independent
-tool/service. In either way, it requires targeting systems(NFVI and
-VIM) are developed/deployed in a way that Escalator could perform
-upgrade on them.
-
-On NFVI system, live-migration is likely used to maintain availability
-because OPNFV would like to make HA transparent from end user. This
-requires VIM system being able to put compute node into maintenance mode
-and then isolated from normal service. Otherwise, new NFVI instances
-might risk at being schedule into the upgrading node.
-
-On VIM system, availability is likely achieved by redundancy. This
-impose less requirements on system/services being upgrade (see PVA
-comments in early version). However, there should be a way to put the
-target system into standby mode. Because starting upgrade on the
-master node in a cluster is likely a bad idea.
-
-.. <hujie>Revised text to be general.
-
-1. It is required for NFVI/VIM to support **service handover** mechanism
- that minimize interruption to 0.001%(i.e. 99.999% service
- availability). Possible implementations are live-migration, redundant
- deployment, etc, (Note: for VIM, interruption could be less
- restrictive)
-
-2. It is required for NFVI/VIM to restore the early version in a efficient
- way, such as **snapshot**.
-
-3. It is required for NFVI/VIM to **migration data** efficiently between
- base and upgraded system.
-
-4. It is recommend for NFV/VIM's interface to support upgrade
- orchestration, e.g. reading/setting system state.
-
-
+++ /dev/null
-Reference Architecture
-----------------------
-
-This section describes the reference architecture, the function blocks,
-and the function entities of Escalator for the reader to well understand how
-the basic functions to be organized.
-
-Upgrade Scope
-~~~~~~~~~~~~~~~~
-
-Upgrade objects described in this document are software programs covered by
-red box in the picture below which includes: VIM and NFVI.
-The target of the upgrade is to reduce the impact on the applications in the
-blue box below as much as possible.
-Note that this upgrade process does not take into consideration the effects
-of Vi-Vnfm and Or-Vi. In other words, the unserviceability of the two
-interfaces during upgrade can be accepted.
-
-.. figure:: images/figure1.png
- :name: figure1
- :width: 100%
-
-The software stack on each node is generally as shown in the table below.
-
-.. figure:: images/figure2.png
- :name: figure2
- :width: 100%
-
-Because the control node upgrade will not affect the VNFs service in the blue
-box, this scheme focuses on upgrading of compute nodes.
-
-Precondition of Upgrade
-~~~~~~~~~~~~~~~~~~~~~~~
-
-1 The environmental requirements
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-1. System is running normally. If there are any faults before the upgrade,
-it is difficult to distinguish between upgrade introduced and the environment
-itself.
-
-2. The environment should have the redundant resources. Because the upgrade
-process is based on the business migration, in the absence of resource
-redundancy,it is impossible to realize the business migration, as well as to
-achieve a smooth upgrade.
-
-Resource redundancy in two levels:
-
-1 NFVI level: This level is mainly the compute nodes resource redundancy.
-During the upgrade, the virtual machine on business can be migrated to another
-free compute node.
-
-2 VNF level: This level depends on HA mechanism in VNF, such as:
-active-standby, load balance. In this case, as long as business of the target
-node on VMs is migrated to other free nodes, the migration of VM might not be
-necessary.
-
-The way of redundancy to be used is subject to the specific environment.
-Generally speaking, During the upgrade, the VNF's service level availability
-mechanism should be used in higher priority than the NFVI's. This will help
-us to reduce the service outage.
-
-2 The requirements for component release version
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-This is primarily a compatibility requirement. You can refer to Linux/Python
-Compatible Semantic Versioning 3.0.0:
-
-Given a version number MAJOR.MINOR.PATCH, increment the:
-
-1. MAJOR version when you make incompatible API changes,
-
-2. MINOR version when you add functionality in a backwards-compatible manner,
-
-3. PATCH version when you make backwards-compatible bug fixes.
-
-Some internal interfaces of OpenStack will be used by Escalator indirectly,
-such as VM migration related interface between VIM and NFVI. So it is required
-to be backward compatible on these interfaces. Refer to "Interface" chapter
-for details.
-
-Upgrade related modules in VIM
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Upgrade operations are initiated by the user through the VIM. For VIM, upgrade
-management mainly contains the object:
-
-- **Upgrade Manager**
-
-Mainly responsible for control of the upgrade process.The Escalator is required
-to know the software deployment information of the platform and will use these
-information during the upgrading. It will be collected from some place, such
-as the Installer, Deploy Manager and Escalator itself, etc.
-
-- **VIM Interface**:
-
-Mainly responsible for the external interface, include Vi-Vnfm, Or-Vi. This
-module stores VNFO and VNFM external information such as address and
-authentication.
-
-- **Cloud Manager**:
-
-Mainly responsible for virtualization resources management,which might be
-considered made up of Openstack and SDN control node.
-
-- **System Support**:
-
-This layer is the runtime support environment of upper layers, e.g. Cloud
-Manager and VIM interface., including:OS, HA, etc. To upgrade the upper
-software is based on this module.
-
-.. figure:: images/figure3.png
- :name: figure3
- :width: 100%
\ No newline at end of file
+++ /dev/null
-Interfaces and Files
---------------------
-
-This section describes the required interfaces and files of Escalator.
-
-
-CLI Interface
-~~~~~~~~~~~~~~~~
-
-This section describes CLI of Escalator.
-
-RESTful API
-~~~~~~~~~~~
-
-This section describes the API of Escalator for developer.
-
-Configuration File
-~~~~~~~~~~~~~~~~~~
-
-This section will suggest a format of the configuration files and how to
-deal with it.
-
-Log File
-~~~~~~~~
-
-This section will suggest a format of the log files and how to deal with
-it.
\ No newline at end of file
+++ /dev/null
-Useful Working Drafts of ETSI NFV
----------------------------------
-
-Access them with your own ETSI account, please DO NOT disclose the
-content.
-
-[1] Migrate Virtualised Compute Resource operation @ 7.3.1.8
-ftp://docbox.etsi.org/ISG/NFV/Open/Drafts/IFA005_Or-Vi_ref_point_Spec/NFV-IFA005v070.zip
-
-[2] Reliability issues during NFV Software upgrade and improvement mechanisms @ 8
-ftp://@docbox.etsi.org/ISG/NFV/Open/Drafts/REL003_E2E_reliability_models/NFV-REL003v030.zip
--- /dev/null
+======================
+Reference Architecture
+======================
+
+This section describes the reference architecture, the function blocks,
+and the function entities of Escalator for the reader to well understand how
+the basic functions to be organized.
+
+The software stack on each node is generally as shown in the table below.
+
+.. figure:: images/figure2.png
+ :name: figure2
+ :width: 100%
+
+Since the upgrading of control node will not affect the VNFs service in the blue
+box, this chapter will focusing on the upgrading of compute nodes.
+
+
+Precondition of Upgrade
+=======================
+
+Upgrade related modules in VIM
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Upgrade operations are initiated by the user through the VIM. For VIM, upgrade
+management mainly contains the object:
+
+- **Upgrade Manager**
+
+Mainly responsible for control of the upgrade process.The Escalator is required
+to know the software deployment information of the platform and will use these
+information during the upgrading. It will be collected from some place, such
+as the Installer, Deploy Manager and Escalator itself, etc.
+
+- **VIM Interface**:
+
+Mainly responsible for the external interface, include Vi-Vnfm, Or-Vi. This
+module stores VNFO and VNFM external information such as address and
+authentication.
+
+- **Cloud Manager**:
+
+Mainly responsible for virtualization resources management,which might be
+considered made up of Openstack and SDN control node.
+
+- **System Support**:
+
+This layer is the runtime support environment of upper layers, e.g. Cloud
+Manager and VIM interface., including:OS, HA, etc. To upgrade the upper
+software is based on this module.
+
+.. figure:: images/figure3.png
+ :name: figure3
+ :width: 100%
+=================
Information Flows
------------------
+=================
This section describes the information flows among the function
entities when Escalator is in actions.
5. After upgraded, Upgrade Manager removes maintenance mode for the compute
node A.
6. Upgrade Manager claims VNFM computing nodes A available.
-7. Select computer node B to upgrade
\ No newline at end of file
+7. Select computer node B to upgrade
--- /dev/null
+=========================
+Administrative Interfaces
+=========================
+
+This section describes the required administrative interfaces of Escalator.
+
+CLI Interface
+=============
+
+This section describes CLI of Escalator.
+
+RESTful API
+===========
+
+This section describes the API of Escalator for developer.
+
--- /dev/null
+=========================
+Configuration and Logging
+=========================
+
+This section describes the required configuration and logging of Escalator.
+
+
+Configuration Format
+====================
+
+This section will suggest a format of the configuration files and how to
+deal with it.
+
+Logging Format
+==============
+
+This section will suggest a format of the log files and how to deal with
+it.
+========
Appendix
---------
+========
A.1 Impact Analysis
-~~~~~~~~~~~~~~~~~~~
+===================
Upgrading the different software modules may cause different impact on
the availability of the infrastructure resources and even on the service
#. Hypervisor, such as KVM, QEMU, XEN, libvirt
#. Openstack agent in computing nodes (like Nova agent, Ceilometer
agent...)
-
+
.. <MT> As SW module, we should list the host OS and maybe its
drivers as well. From upgrade perspective do we limit host OS
upgrades to patches only?
--- /dev/null
+import datetime
+import sys
+import os
+
+try:
+ __import__('imp').find_module('sphinx.ext.numfig')
+ extensions = ['sphinx.ext.numfig']
+except ImportError:
+ # 'pip install sphinx_numfig'
+ extensions = ['sphinx_numfig']
+
+# numfig:
+number_figures = True
+figure_caption_prefix = "Fig."
+
+source_suffix = '.rst'
+master_doc = 'index'
+pygments_style = 'sphinx'
+html_use_index = False
+
+pdf_documents = [('index', u'OPNFV', u'OPNFV Project', u'OPNFV')]
+pdf_fit_mode = "shrink"
+pdf_stylesheets = ['sphinx','kerning','a4']
+#latex_domain_indices = False
+#latex_use_modindex = False
+
+latex_elements = {
+ 'printindex': '',
+}
+
+project = u'OPNFV: Template documentation config'
+copyright = u'%s, OPNFV' % datetime.date.today().year
+version = u'1.0.0'
+release = u'1.0.0'
:alt: OPNFV
:align: left
-ESCALATOR
+ESCALATOR DESIGN CONSIDERATIONS
=======================================
Contents:
:maxdepth: 4
:titlesonly:
- 00-Authors.rst
- 01-Scope.rst
- 02-Background_and_Terminologies.rst
- 03-Functional_Requirements.rst
- 04-Use_Cases_and_Scenarios.rst
- 05-Reference_Architecture.rst
- 06-Information_Flows.rst
- 07-Interfaces_and_Files.rst
- 08-Requirements_from_other_OPNFV_Project.rst
- 09-Reference.rst
- 10-Useful_Working_Drafts_of_ETSI_NFV.rst
- A1-Appendix.rst
+
+
+ 201-Reference_Architecture.rst
+ 202-Information_Flows.rst
+ 203-Administrative_Interfaces.rst
+ 204-Configuration_and_Logging.rst
+ 300-Gap_Analysis_Report.rst
* :ref:`search`
--- /dev/null
+import datetime
+import sys
+import os
+
+try:
+ __import__('imp').find_module('sphinx.ext.numfig')
+ extensions = ['sphinx.ext.numfig']
+except ImportError:
+ # 'pip install sphinx_numfig'
+ extensions = ['sphinx_numfig']
+
+# numfig:
+number_figures = True
+figure_caption_prefix = "Fig."
+
+source_suffix = '.rst'
+master_doc = 'index'
+pygments_style = 'sphinx'
+html_use_index = False
+
+pdf_documents = [('index', u'OPNFV', u'OPNFV Project', u'OPNFV')]
+pdf_fit_mode = "shrink"
+pdf_stylesheets = ['sphinx','kerning','a4']
+#latex_domain_indices = False
+#latex_use_modindex = False
+
+latex_elements = {
+ 'printindex': '',
+}
+
+project = u'OPNFV: Template documentation config'
+copyright = u'%s, OPNFV' % datetime.date.today().year
+version = u'1.0.0'
+release = u'1.0.0'
--- /dev/null
+===============
+Impact Analysis
+===============
+
+Upgrading the different software modules may cause different impact on
+the availability of the infrastructure resources and even on the service
+continuity of the vNFs.
+
+**Software modules in the computing nodes**
+
+#. Host OS patch
+
+#. Hypervisor, such as KVM, QEMU, XEN, libvirt
+#. Openstack agent in computing nodes (like Nova agent, Ceilometer
+ agent...)
+
+.. <MT> As SW module, we should list the host OS and maybe its
+ drivers as well. From upgrade perspective do we limit host OS
+ upgrades to patches only?
+
+**Software modules in network nodes**
+
+#. Neutron L2/L3 agent
+#. OVS, SR-IOV Driver
+
+**Software modules storage nodes**
+
+#. Ceph
+
+The table below analyses such an impact - considering a single instance
+of each software module - from the following aspects:
+
+- the function which will be lost during upgrade,
+- the duration of the loss of this specific function,
+- if this causes the loss of the vNF function,
+- if it causes incompatibility in the different parts of the software,
+- what should be backed up before the upgrade,
+- the duration of restoration time if the upgrade fails
+
+These values provided come from internal testing and based on some
+assumptions, they may vary depending on the deployment techniques.
+Please feel free to add if you find more efficient values during your
+testing.
+
+https://wiki.opnfv.org/_media/upgrade_analysis_v0.5.xlsx
+
+Note that no redundancy of the software modules is considered in the table.
--- /dev/null
+import datetime
+import sys
+import os
+
+try:
+ __import__('imp').find_module('sphinx.ext.numfig')
+ extensions = ['sphinx.ext.numfig']
+except ImportError:
+ # 'pip install sphinx_numfig'
+ extensions = ['sphinx_numfig']
+
+# numfig:
+number_figures = True
+figure_caption_prefix = "Fig."
+
+source_suffix = '.rst'
+master_doc = 'index'
+pygments_style = 'sphinx'
+html_use_index = False
+
+pdf_documents = [('index', u'OPNFV', u'OPNFV Project', u'OPNFV')]
+pdf_fit_mode = "shrink"
+pdf_stylesheets = ['sphinx','kerning','a4']
+#latex_domain_indices = False
+#latex_use_modindex = False
+
+latex_elements = {
+ 'printindex': '',
+}
+
+project = u'OPNFV: Template documentation config'
+copyright = u'%s, OPNFV' % datetime.date.today().year
+version = u'1.0.0'
+release = u'1.0.0'
--- /dev/null
+.. OPNFV Release Engineering documentation, created by
+ sphinx-quickstart on Tue Jun 9 19:12:31 2015.
+ You can adapt this file completely to your liking, but it should at least
+ contain the root `toctree` directive.
+
+.. image:: etc/opnfv-logo.png
+ :height: 40
+ :width: 200
+ :alt: OPNFV
+ :align: left
+
+ESCALATOR GAP ANALYSIS REPORT
+=======================================
+
+Contents:
+
+.. toctree::
+ :maxdepth: 4
+ :titlesonly:
+
+
+
+
+ 301-Impact_Analysis.rst
+
+* :ref:`search`
+
+Revision: _sha1_
+
+Build date: |today|
+++ /dev/null
-See https://wiki.opnfv.org/documentation/tools .
-Authors:
---------
+============
+Contributors
+============
| Jie Hu (ZTE, hu.jie@zte.com.cn)
| Qiao Fu (China Mobile, fuqiao@chinamobile.com)
| Zhipeng Huang (Huawei, huangzhipeng@huawei.com)
| Jia Meng (ZTE, meng.jia@zte.com.cn)
| Liyi Meng (Ericsson, liyi.meng@ericsson.com)
-| Pasi Vaananen (Stratus, pasi.vaananen@stratus.com)
\ No newline at end of file
+| Pasi Vaananen (Stratus, pasi.vaananen@stratus.com)
+=====
Scope
------
+=====
This document describes the user requirements on the smooth upgrade
function of the NFVI and VIM with respect to the upgrades of the OPNFV
(code name: Escalator) with pre-configured data. The upgrade process
includes preparation, validation, execution, monitoring and
conclusion.
-
+
.. <MT> While it is good to have a tool for the entire upgrade process,
but it is a challenging task, so maybe we shouldn't require automation
for the entire process right away. Automation is essential at
execution.
-
+
.. <hujie> Maybe we can analysis information flows of the upgrade tool,
abstract the basic / essential actions from the tool (or tools), and
map them to a command set of NFVI / VIM's interfaces.
The requirements may apply to different NFV functions (NFVI, or VIM, or
both of them). They will be classified in the Appendix of this
-document.
\ No newline at end of file
+document.
+
+The objects being upgraded described in this document are software modules covered by
+red box in the picture below which includes: VIM and NFVI.
+
+The target of the upgrade is to reduce the impact on the applications in the
+blue box below as much as possible.
+
+Please keep in mind that the upgrade tool does not take Vi-Vnfm and Or-Vi into
+consideration. In other words, these two interfaces may not provided service normally
+during upgrade procedure.
+
+
+.. figure:: images/figure1.png
+ :name: figure1
+ :width: 100%
+
--- /dev/null
+===========
+Terminology
+===========
+
+Terminologies
+=============
+
+Operator
+ The term refers to network service providers and Virtual Network
+ Function (VNF) providers.
+
+End-User
+ The term refers to a subscriber of the Operator's services.
+
+Network Service
+ The term refers to a service provided by an Operator to its
+ end-users using a set of (virtualized) Network Functions
+
+Infrastructure Services
+ The term refers to services provided by the NFV Infrastructure to the VNFs
+ as required by the Management & Orchestration functions and especially the VIM.
+ I.e. these are the virtual resources as perceived by the VNFs.
+
+Smooth Upgrade
+ The term refers to an upgrade that results in no service outage
+ for the end-users.
+
+Rolling Upgrade
+ The term refers to an upgrade strategy, which upgrades a node or a subset
+ of nodes at a time in a wave style rolling through the data centre. It
+ is a popular upgrade strategy to maintain service availability.
+
+Parallel Universe Upgrade
+ The term refers to an upgrade strategy, which creates and deploys
+ a new universe - a system with the new configuration - while the old
+ system continues running. The state of the old system is transferred
+ to the new system after sufficient testing of the new system.
+
+Infrastructure Resource Model
+ The term refers to the representation of infrastructure resources,
+ namely: the physical resources, the virtualization
+ facility resources and the virtual resources.
+
+Physical Resource
+ The term refers to a piece of hardware in the NFV infrastructure that may
+ also include firmware enabling this piece of hardware.
+
+Virtual Resource
+ The term refers to a resource, which is provided as services built on top
+ of the physical resources via the virtualization facilities; in particular,
+ virtual resources are the resources on which VNFs are deployed. Examples of
+ virtual resources are: VMs, virtual switches, virtual routers, virtual disks.
+
+Visualization Facility
+ The term refers to a resource that enables the creation
+ of virtual environments on top of the physical resources, e.g.
+ hypervisor, OpenStack, etc.
+
+Upgrade Campaign
+ The term refers to a choreography that describes how the upgrade should
+ be performed in terms of its targets (i.e. upgrade objects), the
+ steps/actions required of upgrading each, and the coordination of these
+ steps so that service availability can be maintained. It is an input to an
+ upgrade tool (Escalator) to carry out the upgrade.
+
+Upgrade Duration
+ The duration of an upgrade characterized by the time elapsed between its
+ initiation and its completion. E.g. from the moment the execution of an
+ upgrade campaign has started until it has been committed. Depending on
+ the upgrade strategy, the state of the configuration and the upgrade target
+ some parts of the system may be in a more vulnerable state with respect to
+ service availbility.
+
+Outage
+ The period of time during which a given service is not provided is referred
+ as the outage of that given service. If a subsystem or the entire system
+ does not provide any service, it is the outage of the given subsystem or the
+ system. Smooth upgrade means upgrade with no outage for the user plane, i.e.
+ no VNF should experience service outage.
+
+Rollback
+ The term refers to a failure handling strategy that reverts the changes
+ done by a potentially failed upgrade execution one by one in a reverse order.
+ I.e. it is like undoing the changes done by the upgrade.
+
+Backup
+ The term refers to data persisted to a storage, so that it can be used to
+ restore the system or a given part of it in the same state as it was when the
+ backup was created assuming a cold restart. Changes made to the system from
+ the moment the backup was created till the moment it is used to restore the
+ (sub)system are lost in the restoration process.
+
+Restore
+ The term refers to a failure handling strategy that reverts the changes
+ done, for example, by an upgrade by restoring the system from some backup
+ data. This results in the loss of any change and data persisted after the
+ backup was been taken. To recover those additional measures need to be taken
+ if necessary (e.g. rollforward).
+
+Rollforward
+ The term refers to a failure handling strategy applied after a restore
+ (from a backup) opertaion to recover any loss of data persisted between
+ the time the backup has been taken and the moment it is restored. Rollforward
+ requires that data that needs to survive the restore operation is logged at
+ a location not impacted by the restore so that it can be re-applied to the
+ system after its restoration from the backup.
+
+Downgrade
+ The term refers to an upgrade in which an earlier version of the software
+ is restored through the upgrade procedure. A system can be downgraded to any
+ earlier version and the compatibility of the versions will determine the
+ applicable upgrade strategies and whether service outage can be avoided.
+ In particular any data conversion needs special attention.
+
+Abbreviations
+=============
+
+NFVI
+ The term is an abbreviation for Network Function Virtualization
+ Infrastructure; sometimes it is also referred as data plane in this
+ document. The NFVI provides the virtual resources to the virtual
+ network functions under the control of the VIM.
+
+VIM
+ The term is an abbreviation for Virtual Infrastructure Manager;
+ sometimes it is also referred as control plane in this document.
+ The VIM controls and manages the NFVI compute, network and storage
+ resources to provide the required virtual resources to the VNFs.
+
--- /dev/null
+==========
+Background
+==========
+
+Upgrade Objects
+===============
+
+Physical Resource
+^^^^^^^^^^^^^^^^^
+
+Most cloud infrastructures support the dynamic addition and removal of
+hardware. Accordingly a hardware upgrade could be done by adding the new
+piece of hardware and removing the old one. From the persepctive of smooth
+upgrade the orchestration/scheduling of these actions is the primary concern.
+
+Upgrading a physical resource may involve as well the upgrade of its firmware
+and/or modifying its configuration data. This may require the restart of the
+hardware.
+
+Virtual Resources
+^^^^^^^^^^^^^^^^^
+
+Addition and removal of virtual resources may be initiated by the users or be
+a result of an elasticity action. Users may also request the upgrade of their
+virtual resources using a new VM image.
+
+.. Needs to be moved to requirement section: Escalator should facilitate such an
+ option and allow for a smooth upgrade.
+
+On the other hand changes in the infrastructure, namely, in the hardware and/or
+the virtualization facility resources may result in the upgrade of the virtual
+resources. For example if by some reason the hypervisor is changed and
+the current VMs cannot be migrated to the new hypervisor - they are
+incompatible - then the VMs need to be upgraded too. This is not
+something the NFVI user (i.e. VNFs ) would know about.
+
+
+Virtualization Facility Resources
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Based on the functionality they provide, virtualization facility
+resources could be divided into computing node, networking node,
+storage node and management node.
+
+The possible upgrade objects in these nodes are considered below:
+(Note: hardware based virtualization may be considered as virtualization
+facility resource, but from escalator perspective, it is better to
+consider it as part of the hardware upgrade. )
+
+**Computing node**
+
+1. OS Kernel
+
+2. Hypvervisor and virtual switch
+
+3. Other kernel modules, like drivers
+
+4. User space software packages, like nova-compute agents and other
+ control plane programs.
+
+Updating 1 and 2 will cause the loss of virtualzation functionality of
+the compute node, which may lead to the interruption of data plane services
+if the virtual resource is not redudant.
+
+Updating 3 might have the same result.
+
+Updating 4 might lead to control plane services interruption if not an
+HA deployment.
+
+.. <MT> I'm not sure why would 4 cause control plane interruption on a
+ compute node. My understanding is that simply the node cannot be managed.
+ Redundancy won't help in that either.
+
+
+**Networking node**
+
+1. OS kernel, optional, not all switches/routers allow the upgrade their
+ OS since it is more like a firmware than a generic OS.
+
+2. User space software package, like neutron agents and other control
+ plane programs
+
+Updating 1 if allowed will cause a node reboot and therefore leads to
+data plane service interruption if the virtual resource is not
+redundant.
+
+Updating 2 might lead to control plane services interruption if not an
+HA deployment.
+
+**Storage node**
+
+1. OS kernel, optional, not all storage nodes allow the upgrade their OS
+ since it is more like a firmware than a generic OS.
+
+2. Kernel modules
+
+3. User space software packages, control plane programs
+
+Updating 1 if allowed will cause a node reboot and therefore leads to
+data plane services interruption if the virtual resource is not
+redundant.
+
+Update 2 might result in the same.
+
+Updating 3 might lead to control plane services interruption if not an
+HA deployment.
+
+**Management node**
+
+1. OS Kernel
+
+2. Kernel modules, like driver
+
+3. User space software packages, like database, message queue and
+ control plane programs.
+
+Updating 1 will cause a node reboot and therefore leads to control
+plane services interruption if not an HA deployment. Updating 2 might
+result in the same.
+
+Updating 3 might lead to control plane services interruption if not an
+HA deployment.
+
+Upgrade Granularity
+===================
+
+The granularity of an upgrade can be characterized from two perspective:
+- the physical dimension and
+- the software dimension
+
+Physical Dimension
+^^^^^^^^^^^^^^^^^^
+
+The physical dimension characterizes the number of similar upgrade objects
+targeted by the upgrade, i.e. whether it is full / partial upgrade of a
+data centre, cluster, zone.
+Because of the upgrade of a data centre or a zone, it may be divided into
+several batches. Thus there is a need for efficiency in the execution of
+upgrades of potentially huge number of upgrade objects while still maintain
+availability to fulfill the requirement of smooth upgrade.
+
+The upgrade of a cloud environment (cluster) may also
+be partial. For example, in one cloud environment running a number of
+VNFs, we may just try to upgrade one of them to check the stability and
+performance, before we upgrade all of them.
+Thus there is a need for proper organization of the artifacts associated with
+the different upgrade objects. Also the different versions should be able
+to coextist beyond the upgrade period.
+
+From this perspective special attention may be needed when upgrading
+objects that are collaborating in a redundancy schema as in this case
+different versions not only need to coexist but also collaborate. This
+puts requirement on the upgrade objects primarily. If this is not possible
+the upgrade campaign should be designed in such a way that the proper
+isolation is ensured.
+
+Software Dimension
+^^^^^^^^^^^^^^^^^^
+
+The software dimension of the upgrade characterizes the upgrade object
+type targeted and the combination in which they are upgraded together.
+
+Even though the upgrade may
+initially target only one type of upgrade object, e.g. the hypervisor
+the dependency of other upgrade objects on this initial target object may
+require their upgrade as well. I.e. the upgrades need to be combined. From this
+perspective the main concern is compatibility of the dependent and
+sponsor objects. To take into consideration of these dependencies
+they need to be described together with the version compatility information.
+Breaking dependencies is the major cause of outages during upgrades.
+
+In other cases it is more efficient to upgrade a combination of upgrade
+objects than to do it one by one. One aspect of the combination is how
+the upgrade packages can be combined, whether a new image can be created for
+them before hand or the different packages can be installed during the upgrade
+independently, but activated together.
+
+The combination of upgrade objects may span across
+layers (e.g. software stack in the host and the VM of the VNF).
+Thus, it may require additional coordination between the management layers.
+
+With respect to each upgrade object type and even stacks we can
+distingush major and minor upgrades:
+
+**Major Upgrade**
+
+Upgrades between major releases may introducing significant changes in
+function, configuration and data, such as the upgrade of OPNFV from
+Arno to Brahmaputra.
+
+**Minor Upgrade**
+
+Upgrades inside one major releases which would not leads to changing
+the structure of the platform and may not infect the schema of the
+system data.
+
+Scope of Impact
+===============
+
+Considering availability and therefore smooth upgrade, one of the major
+concerns is the predictability and control of the outcome of the different
+upgrade operations. Ideally an upgrade can be performed without impacting any
+entity in the system, which means none of the operations change or potentially
+change the behaviour of any entity in the system in an uncotrolled manner.
+Accordingly the operations of such an upgrade can be performed any time while
+the system is running, while all the entities are online. No entity needs to be
+taken offline to avoid such adverse effects. Hence such upgrade operations
+are referred as online operations. The effects of the upgrade might be activated
+next time it is used, or may require a special activation action such as a
+restart. Note that the activation action provides more control and predictability.
+
+If an entity's behavior in the system may change due to the upgrade it may
+be better to take it offline for the time of the relevant upgrade operations.
+The main question is however considering the hosting relation of an upgrade
+object what hosted entities are impacted. Accordingly we can identify a scope
+which is impacted by taking the given upgrade object offline. The entities
+that are in the scope of impact may need to be taken offline or moved out of
+this scope i.e. migrated.
+
+If the impacted entity is in a different layer managed by another manager
+this may require coordination because taking out of service some
+infrastructure resources for the time of their upgrade which support virtual
+resources used by VNFs that should not experience outages. The hosted VNFs
+may or may not allow for the hot migration of their VMs. In case of migration
+the VMs placement policy should be considered.
+
--- /dev/null
+============
+Requirements
+============
+
+Upgrade duration
+================
+
+As the OPNFV end-users are primarily Telecom operators, the network
+services provided by the VNFs deployed on the NFVI should meet the
+requirement of 'Carrier Grade'.::
+
+ In telecommunication, a "carrier grade" or"carrier class" refers to a
+ system, or a hardware or software component that is extremely reliable,
+ well tested and proven in its capabilities. Carrier grade systems are
+ tested and engineered to meet or exceed "five nines" high availability
+ standards, and provide very fast fault recovery through redundancy
+ (normally less than 50 milliseconds). [from wikipedia.org]
+
+"five nines" means working all the time in ONE YEAR except 5'15".
+
+::
+
+ We have learnt that a well prepared upgrade of OpenStack needs 10
+ minutes. The major time slot in the outage time is used spent on
+ synchronizing the database. [from ' Ten minutes OpenStack Upgrade? Done!
+ ' by Symantec]
+
+This 10 minutes of downtime of the OpenStack services however did not impact the
+users, i.e. the VMs running on the compute nodes. This was the outage of
+the control plane only. On the other hand with respect to the
+preparations this was a manually tailored upgrade specific to the
+particular deployment and the versions of each OpenStack service.
+
+The project targets to achieve a more generic methodology, which however
+requires that the upgrade objects fulfil certain requirements. Since
+this is only possible on the long run we target first the upgrade
+of the different VIM services from version to version.
+
+**Questions:**
+
+1. Can we manage to upgrade OPNFV in only 5 minutes?
+
+.. <MT> The first question is whether we have the same carrier grade
+ requirement on the control plane as on the user plane. I.e. how
+ much control plane outage we can/willing to tolerate?
+ In the above case probably if the database is only half of the size
+ we can do the upgrade in 5 minutes, but is that good? It also means
+ that if the database is twice as much then the outage is 20
+ minutes.
+ For the user plane we should go for less as with two release yearly
+ that means 10 minutes outage per year.
+
+.. <Malla> 10 minutes outage per year to the users? Plus, if we take
+ control plane into the consideration, then total outage will be
+ more than 10 minute in whole network, right?
+
+.. <MT> The control plane outage does not have to cause outage to
+ the users, but it may of course depending on the size of the system
+ as it's more likely that there's a failure that needs to be handled
+ by the control plane.
+
+2. Is it acceptable for end users ? Such as a planed service
+ interruption will lasting more than ten minutes for software
+ upgrade.
+
+.. <MT> For user plane, no it's not acceptable in case of
+ carrier-grade. The 5' 15" downtime should include unplanned and
+ planned downtimes.
+
+.. <Malla> I go agree with Maria, it is not acceptable.
+
+3. Will any VNFs still working well when VIM is down?
+
+.. <MT> In case of OpenStack it seems yes. .:)
+
+The maximum duration of an upgrade
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The duration of an upgrade is related to and proportional with the
+scale and the complexity of the OPNFV platform as well as the
+granularity (in function and in space) of the upgrade.
+
+.. <Malla> Also, if is a partial upgrade like module upgrade, it depends
+ also on the OPNFV modules and their tight connection entities as well.
+
+.. <MT> Since the maintenance window is shrinking and becoming non-existent
+ the duration of the upgrade is secondary to the requirement of smooth upgrade.
+ But probably we want to be able to put a time constraint on each upgrade
+ during which it must complete otherwise it is considered failed and the system
+ should be rolled back. I.e. in case of automatic execution it might not be clear
+ if an upgrade is long or just hanging. The time constraints may be a function
+ of the size of the system in terms of the upgrade object(s).
+
+The maximum duration of a roll back when an upgrade is failed
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The duration of a roll back is short than the corresponding upgrade. It
+depends on the duration of restore the software and configure data from
+pre-upgrade backup / snapshot.
+
+.. <MT> During the upgrade process two types of failure may happen:
+ In case we can recover from the failure by undoing the upgrade
+ actions it is possible to roll back the already executed part of the
+ upgrade in graceful manner introducing no more service outage than
+ what was introduced during the upgrade. Such a graceful roll back
+ requires typically the same amount of time as the executed portion of
+ the upgrade and impose minimal state/data loss.
+
+.. <MT> Requirement: It should be possible to roll back gracefully the
+ failed upgrade of stateful services of the control plane.
+ In case we cannot recover from the failure by just undoing the
+ upgrade actions, we have to restore the upgraded entities from their
+ backed up state. In other terms the system falls back to an earlier
+ state, which is typically a faster recovery procedure than graceful
+ roll back and depending on the statefulness of the entities involved it
+ may result in significant state/data loss.
+
+.. <MT> Two possible types of failures can happen during an upgrade
+
+.. <MT> We can recover from the failure that occurred in the upgrade process:
+ In this case, a graceful rolling back of the executed part of the
+ upgrade may be possible which would "undo" the executed part in a
+ similar fashion. Thus, such a roll back introduces no more service
+ outage during an upgrade than the executed part introduced. This
+ process typically requires the same amount of time as the executed
+ portion of the upgrade and impose minimal state/data loss.
+
+.. <MT> We cannot recover from the failure that occurred in the upgrade
+ process: In this case, the system needs to fall back to an earlier
+ consistent state by reloading this backed-up state. This is typically
+ a faster recovery procedure than the graceful roll back, but can cause
+ state/data loss. The state/data loss usually depends on the
+ statefulness of the entities whose state is restored from the backup.
+
+The maximum duration of a VNF interruption (Service outage)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Since not the entire process of a smooth upgrade will affect the VNFs,
+the duration of the VNF interruption may be shorter than the duration
+of the upgrade. In some cases, the VNF running without the control
+from of the VIM is acceptable.
+
+.. <MT> Should require explicitly that the NFVI should be able to
+ provide its services to the VNFs independent of the control plane?
+
+.. <MT> Requirement: The upgrade of the control plane must not cause
+ interruption of the NFVI services provided to the VNFs.
+
+.. <MT> With respect to carrier-grade the yearly service outage of the
+ VNF should not exceed 5' 15" regardless whether it is planned or
+ unplanned outage. Considering the HA requirements TL-9000 requires an
+ end-to-end service recovery time of 15 seconds based on which the ETSI
+ GS NFV-REL 001 V1.1.1 (2015-01) document defines three service
+ availability levels (SAL). The proposed example service recovery times
+ for these levels are:
+
+.. <MT> SAL1: 5-6 seconds
+
+.. <MT> SAL2: 10-15 seconds
+
+.. <MT> SAL3: 20-25 seconds
+
+.. <Pva> my comment was actually that the downtime metrics of the
+ underlying elements, components and services are small fraction of the
+ total E2E service availability time. No-one on the E2E service path
+ will get the whole downtime allocation (in this context it includes
+ upgrade process related outages for the services provided by VIM etc.
+ elements that are subject to upgrade process).
+
+.. <MT> So what you are saying is that the upgrade of any entity
+ (component, service) shouldn't cause even this much service
+ interruption. This was the reason I brought these figures here as well
+ that they are posing some kind of upper-upper boundary. Ideally the
+ interruption is in the millisecond range i.e. no more than a
+ switch-over or a live migration.
+
+.. <MT> Requirement: Any interruption caused to the VNF by the upgrade
+ of the NFVI should be in the sub-second range.
+
+.. <MT]> In the future we also need to consider the upgrade of the NFVI,
+ i.e. HW, firmware, hypervisors, host OS etc.
+
+Pre-upgrading Environment
+=========================
+
+System is running normally. If there are any faults before the upgrade,
+it is difficult to distinguish between upgrade introduced and the environment
+itself.
+
+The environment should have the redundant resources. Because the upgrade
+process is based on the business migration, in the absence of resource
+redundancy,it is impossible to realize the business migration, as well as to
+achieve a smooth upgrade.
+
+Resource redundancy in two levels:
+
+NFVI level: This level is mainly the compute nodes resource redundancy.
+During the upgrade, the virtual machine on business can be migrated to another
+free compute node.
+
+VNF level: This level depends on HA mechanism in VNF, such as:
+active-standby, load balance. In this case, as long as business of the target
+node on VMs is migrated to other free nodes, the migration of VM might not be
+necessary.
+
+The way of redundancy to be used is subject to the specific environment.
+Generally speaking, During the upgrade, the VNF's service level availability
+mechanism should be used in higher priority than the NFVI's. This will help
+us to reduce the service outage.
+
+Release version of software components
+======================================
+
+This is primarily a compatibility requirement. You can refer to Linux/Python
+Compatible Semantic Versioning 3.0.0:
+
+Given a version number MAJOR.MINOR.PATCH, increment the:
+
+MAJOR version when you make incompatible API changes,
+
+MINOR version when you add functionality in a backwards-compatible manner,
+
+PATCH version when you make backwards-compatible bug fixes.
+
+Some internal interfaces of OpenStack will be used by Escalator indirectly,
+such as VM migration related interface between VIM and NFVI. So it is required
+to be backward compatible on these interfaces. Refer to "Interface" chapter
+for details.
+
+Work Flows
+==========
+
+Describes the different types of requirements. To have a table to label the source of
+the requirements, e.g. Doctor, Multi-site, etc.
+
+Basic Actions
+=============
+
+This section describes the basic functions may required by Escalator.
+
+Preparation (offline)
+^^^^^^^^^^^^^^^^^^^^^
+
+This is the design phase when the upgrade plan (or upgrade campaign) is
+being designed so that it can be executed automatically with minimal
+service outage. It may include the following work:
+
+1. Check the dependencies of the software modules and their impact,
+ backward compatibilities to figure out the appropriate upgrade method
+ and ordering.
+2. Find out if a rolling upgrade could be planned with several rolling
+ steps to avoid any service outage due to the upgrade some
+ parts/services at the same time.
+3. Collect the proper version files and check the integration for
+ upgrading.
+4. The preparation step should produce an output (i.e. upgrade
+ campaign/plan), which is executable automatically in an NFV Framework
+ and which can be validated before execution.
+
+ - The upgrade campaign should not be referring to scalable entities
+ directly, but allow for adaptation to the system configuration and
+ state at any given moment.
+ - The upgrade campaign should describe the ordering of the upgrade
+ of different entities so that dependencies, redundancies can be
+ maintained during the upgrade execution
+ - The upgrade campaign should provide information about the
+ applicable recovery procedures and their ordering.
+ - The upgrade campaign should consider information about the
+ verification/testing procedures to be performed during the upgrade
+ so that upgrade failures can be detected as soon as possible and
+ the appropriate recovery procedure can be identified and applied.
+ - The upgrade campaign should provide information on the expected
+ execution time so that hanging execution can be identified
+ - The upgrade campaign should indicate any point in the upgrade when
+ coordination with the users (VNFs) is required.
+
+.. <hujie> Depends on the attributes of the object being upgraded, the
+ upgrade plan may be slitted into step(s) and/or sub-plan(s), and even
+ more small sub-plans in design phase. The plan(s) or sub-plan(s) my
+ include step(s) or sub-plan(s).
+
+Validation the upgrade plan / Checking the pre-requisites of System( offline / online)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The upgrade plan should be validated before the execution by testing
+it in a test environment which is similar to the product environment.
+
+.. <MT> However it could also mean that we can identify some properties
+ that it should satisfy e.g. what operations can or cannot be executed
+ simultaneously like never take out two VMs of the same VNF.
+
+.. <MT> Another question is if it requires that the system is in a particular
+ state when the upgrade is applied. I.e. if there's certain amount of
+ redundancy in the system, migration is enabled for VMs, when the NFVI
+ is upgraded the VIM is healthy, when the VIM is upgraded the NFVI is
+ healthy, etc.
+
+.. <MT> I'm not sure what online validation means: Is it the validation of the
+ upgrade plan/campaign or the validation of the system that it is in a
+ state that the upgrade can be performed without too much risk?==
+
+Before the upgrade plan being executed, the system healthy of the
+online product environment should be checked and confirmed to satisfy
+the requirements which were described in the upgrade plan. The
+sysinfo, e.g. which included system alarms, performance statistics and
+diagnostic logs, will be collected and analogized. It is required to
+resolve all of the system faults or exclude the unhealthy part before
+executing the upgrade plan.
+
+
+Backup/Snapshot (online)
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+For avoid loss of data when a unsuccessful upgrade was encountered, the
+data should be back-upped and the system state snapshot should be taken
+before the execution of upgrade plan. This would be considered in the
+upgrade plan.
+
+Several backups/Snapshots may be generated and stored before the single
+steps of changes. The following data/files are required to be
+considered:
+
+1. running version files for each node.
+2. system components' configuration file and database.
+3. image and storage, if it is necessary.
+
+.. <MT> Does 3 imply VNF image and storage? I.e. VNF state and data?==
+
+.. <hujie> The following text is derived from previous "4. Negotiate
+ with the VNF if it's ready for the upgrade"
+
+Although the upper layer, which include VNFs and VNFMs, is out of the
+scope of Escalator, but it is still recommended to let it ready for a
+smooth system upgrade. The escalator could not guarantee the safe of
+VNFs. The upper layer should have some safe guard mechanism in design,
+and ready for avoiding failure in system upgrade.
+
+Execution (online)
+^^^^^^^^^^^^^^^^^^
+
+The execution of upgrade plan should be a dynamical procedure which is
+ controlled by Escalator.
+
+.. <hujie> Revised text to be general.==
+
+1. It is required to supporting execution ether in sequence or in
+ parallel.
+2. It is required to check the result of the execution and take the
+ action according the situation and the policies in the upgrade plan.
+3. It is required to execute properly on various configurations of
+ system object. I.e. stand-alone, HA, etc.
+4. It is required to execute on the designated different parts of the
+ system. I.e. physical server, virtualized server, rack, chassis,
+ cluster, even different geographical places.
+
+Testing (online)
+^^^^^^^^^^^^^^^^
+
+The testing after upgrade the whole system or parts of system to make
+sure the upgraded system(object) is working normally.
+
+.. <hujie> Revised text to be general.
+
+1. It is recommended to run the prepared test cases to see if the
+ functionalities are available without any problem.
+2. It is recommended to check the sysinfo, e.g. system alarms,
+ performance statistics and diagnostic logs to see if there are any
+ abnormal.
+
+Restore/Roll-back (online)
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When upgrade is failure unfortunately, a quick system restore or system
+roll-back should be taken to recovery the system and the services.
+
+.. <hujie> Revised text to be general.
+
+1. It is recommend to support system restore from backup when upgrade
+ was failed.
+2. It is recommend to support graceful roll-back with reverse order
+ steps if possible.
+
+Monitoring (online)
+^^^^^^^^^^^^^^^^^^^
+
+Escalator should continually monitor the process of upgrade. It is
+keeping update status of each module, each node, each cluster into a
+status table during upgrade.
+
+.. <hujie> Revised text to be general.
+
+1. It is required to collect the status of every objects being upgraded
+ and sending abnormal alarms during the upgrade.
+2. It is recommend to reuse the existing monitoring system, like alarm.
+3. It is recommend to support pro-actively query.
+4. It is recommend to support passively wait for notification.
+
+**Two possible ways for monitoring:**
+
+**Pro-Actively Query** requires NFVI/VIM provides proper API or CLI
+interface. If Escalator serves as a service, it should pass on these
+interfaces.
+
+**Passively Wait for Notification** requires Escalator provides
+callback interface, which could be used by NFVI/VIM systems or upgrade
+agent to send back notification.
+
+.. <hujie> I am not sure why not to subscribe the notification.
+
+Logging (online)
+^^^^^^^^^^^^^^^^
+
+Record the information generated by escalator into log files. The log
+file is used for manual diagnostic of exceptions.
+
+1. It is required to support logging.
+2. It is recommended to include time stamp, object id, action name,
+ error code, etc.
+
+Administrative Control (online)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Administrative Control is used for control the privilege to start any
+escalator's actions for avoiding unauthorized operations.
+
+#. It is required to support administrative control mechanism
+#. It is recommend to reuse the system's own secure system.
+#. It is required to avoid conflicts when the system's own secure system
+ being upgraded.
+
+Requirements on Object being upgraded
+=====================================
+
+.. <hujie> We can develop BPs in future from requirements of this section and
+ gap analysis for upper stream projects
+
+Escalator focus on smooth upgrade. In practical implementation, it
+might be combined with installer/deplorer, or act as an independent
+tool/service. In either way, it requires targeting systems(NFVI and
+VIM) are developed/deployed in a way that Escalator could perform
+upgrade on them.
+
+On NFVI system, live-migration is likely used to maintain availability
+because OPNFV would like to make HA transparent from end user. This
+requires VIM system being able to put compute node into maintenance mode
+and then isolated from normal service. Otherwise, new NFVI instances
+might risk at being schedule into the upgrading node.
+
+On VIM system, availability is likely achieved by redundancy. This
+impose less requirements on system/services being upgrade (see PVA
+comments in early version). However, there should be a way to put the
+target system into standby mode. Because starting upgrade on the
+master node in a cluster is likely a bad idea.
+
+.. <hujie>Revised text to be general.
+
+1. It is required for NFVI/VIM to support **service handover** mechanism
+ that minimize interruption to 0.001%(i.e. 99.999% service
+ availability). Possible implementations are live-migration, redundant
+ deployment, etc, (Note: for VIM, interruption could be less
+ restrictive)
+
+2. It is required for NFVI/VIM to restore the early version in a efficient
+ way, such as **snapshot**.
+
+3. It is required for NFVI/VIM to **migration data** efficiently between
+ base and upgraded system.
+
+4. It is recommend for NFV/VIM's interface to support upgrade
+ orchestration, e.g. reading/setting system state.
+
+Functional Requirements
+=======================
+
+Availability mechanism, etc.
+
+Non-functional Requirements
+===========================
-Use Cases and Scenarios
------------------------
+=========
+Use Cases
+=========
-This section describes the use cases and scenarios to verify the
-requirements of Escalator.
+This section describes the use cases in different system configuration
+to verify the requirements of Escalator.
-Scenarios
-~~~~~~~~~
-1. Upgrade a system with HA configuration
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+System Configurations
+=====================
+
+HA configuration
+^^^^^^^^^^^^^^^^
A HA configuration system is very popular in the operator's data centre.
It is a typical product environment. It is always running 7\*24 with VNFs
running on it to provide services to the end users.
-2. Upgrade a system with non-HA configuration
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Non-HA configuration
+^^^^^^^^^^^^^^^^^^^^
A non-HA configuration system is normally deployed for experimental or
development usages, such as a Vagrant/VM environment.
-Escalator supports the upgrade in this scenario, but it does not guarantee a
-smooth upgrade.
+Escalator supports the upgrade system in this configuration, but it may
+not guarantee a smooth upgrade.
Use cases
-~~~~~~~~~
+=========
+
Use case #1: Smooth upgrade in a HA configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
For a system with HA configuration, the operator can use Escalator to
- Post-Conditions
1. The system is rolled-back successfully when the upgrade failed.
-
+=========
Reference
----------
+=========
[1] ETSI GS NFV 002 (V1.1.1): “Architectural Framework”
-Requirements from other OPNFV projects
---------------------------------------
-
-We have created a questionnaire_ for collecting other projects requirements.
-Please advertise it.
-
-.. _questionnaire: https://docs.google.com/forms/d/11o1mt15zcq0WBtXYK0n6lKF8XuIzQTwvv8ePTjmcoF0/viewform?usp=send_form
-
-
+================================
+Requirements from other Projects
+================================
Doctor Project
-~~~~~~~~~~~~~~
+==============
.. <Malla> This scenario could be out of scope in Escalator project, but
having the option to support this should be better to align with
Doctor requirements.
-
+
The scope of Doctor project also covers maintenance scenario in which
1. The VIM administrator requests host maintenance to VIM.
maintenance" message from the owner (VNFM)
HA Project
-~~~~~~~~~~
+==========
Multi-site Project
-~~~~~~~~~~~~~~~~~~
+==================
- Escalator upgrade one site should at least not lead to the other site
API token validation failed.
--- /dev/null
+==========================
+Questionnaire of Escalator
+==========================
+
+A Questionnaire was created for collecting requirements from other projects.
+
+Escalator Questionnaire:
+https://wiki.opnfv.org/_media/wiki/opnfv_escalator_questionnaire_20150723.pptx
+
+Answer the questionnaire: https://docs.google.com/forms/d/11o1mt15zcq0WBtXYK0n6lKF8XuIzQTwvv8ePTjmcoF0/viewform?usp=send_form
+
--- /dev/null
+===================
+Gap Analysis Report
+===================
+
+Impact Analysis
+===============
+
+Upgrading the different software modules may cause different impact on
+the availability of the infrastructure resources and even on the service
+continuity of the vNFs.
+
+**Software modules in the computing nodes**
+
+#. Host OS patch
+
+#. Hypervisor, such as KVM, QEMU, XEN, libvirt
+#. Openstack agent in computing nodes (like Nova agent, Ceilometer
+ agent...)
+
+.. <MT> As SW module, we should list the host OS and maybe its
+ drivers as well. From upgrade perspective do we limit host OS
+ upgrades to patches only?
+
+**Software modules in network nodes**
+
+#. Neutron L2/L3 agent
+#. OVS, SR-IOV Driver
+
+**Software modules storage nodes**
+
+#. Ceph
+
+The table below analyses such an impact - considering a single instance
+of each software module - from the following aspects:
+
+- the function which will be lost during upgrade,
+- the duration of the loss of this specific function,
+- if this causes the loss of the vNF function,
+- if it causes incompatibility in the different parts of the software,
+- what should be backed up before the upgrade,
+- the duration of restoration time if the upgrade fails
+
+These values provided come from internal testing and based on some
+assumptions, they may vary depending on the deployment techniques.
+Please feel free to add if you find more efficient values during your
+testing.
+
+https://wiki.opnfv.org/_media/upgrade_analysis_v0.5.xlsx
+
+Note that no redundancy of the software modules is considered in the table.
--- /dev/null
+import datetime
+import sys
+import os
+
+try:
+ __import__('imp').find_module('sphinx.ext.numfig')
+ extensions = ['sphinx.ext.numfig']
+except ImportError:
+ # 'pip install sphinx_numfig'
+ extensions = ['sphinx_numfig']
+
+# numfig:
+number_figures = True
+figure_caption_prefix = "Fig."
+
+source_suffix = '.rst'
+master_doc = 'index'
+pygments_style = 'sphinx'
+html_use_index = False
+
+pdf_documents = [('index', u'OPNFV', u'OPNFV Project', u'OPNFV')]
+pdf_fit_mode = "shrink"
+pdf_stylesheets = ['sphinx','kerning','a4']
+#latex_domain_indices = False
+#latex_use_modindex = False
+
+latex_elements = {
+ 'printindex': '',
+}
+
+project = u'OPNFV: Template documentation config'
+copyright = u'%s, OPNFV' % datetime.date.today().year
+version = u'1.0.0'
+release = u'1.0.0'
--- /dev/null
+.. OPNFV Release Engineering documentation, created by
+ sphinx-quickstart on Tue Jun 9 19:12:31 2015.
+ You can adapt this file completely to your liking, but it should at least
+ contain the root `toctree` directive.
+
+.. image:: etc/opnfv-logo.png
+ :height: 40
+ :width: 200
+ :alt: OPNFV
+ :align: left
+
+ESCALATOR USER REQUIREMENTS
+=======================================
+
+Contents:
+
+.. toctree::
+ :maxdepth: 4
+ :titlesonly:
+
+
+ 000-Contributors.rst
+ 101-Scope.rst
+ 102-Terminologies.rst
+ 103-Background.rst
+ 104-Requirements.rst
+ 105-Use_Cases.rst
+ 106-Reference.rst
+ 1A1-Requirements_from_other_Projects.rst
+ 1A2-Questionnaire_of_Escalator.rst
+
+
+* :ref:`search`
+
+Revision: _sha1_
+
+Build date: |today|
Code Review / escalator.git / commitdiff
