--- /dev/null
+General Requirements Background and Terminology\r
+-----------------------------------------------\r
+\r
+Terminologies and definitions\r
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r
+\r
+NFVI\r
+ The term is an abbreviation for Network Function Virtualization\r
+ Infrastructure; sometimes it is also referred as data plane in this\r
+ document.\r
+\r
+VIM\r
+ The term is an abbreviation for Virtual Infrastructure Management;\r
+ sometimes it is also referred as control plane in this document.\r
+\r
+Operator\r
+ The term refers to network service providers and Virtual Network\r
+ Function (VNF) providers.\r
+\r
+End-User\r
+ The term refers to a subscriber of the Operator's services.\r
+\r
+Network Service\r
+ The term refers to a service provided by an Operator to its\r
+ End-users using a set of (virtualized) Network Functions\r
+\r
+Infrastructure Services\r
+ The term refers to services provided by the NFV Infrastructure and the\r
+ the Management & Orchestration functions to the VNFs. I.e.\r
+ these are the virtual resources as perceived by the VNFs.\r
+\r
+Smooth Upgrade\r
+ The term refers to an upgrade that results in no service outage\r
+ for the end-users.\r
+\r
+Rolling Upgrade\r
+ The term refers to an upgrade strategy that upgrades each node or\r
+ a subset of nodes in a wave style rolling through the data centre. It\r
+ is a popular upgrade strategy to maintain service availability.\r
+\r
+Parallel Universe Upgrade\r
+ The term refers to an upgrade strategy that creates and deploys\r
+ a new universe - a system with the new configuration - while the old\r
+ system continues running. The state of the old system is transferred\r
+ to the new system after sufficient testing of the new system.\r
+\r
+Infrastructure Resource Model\r
+ The term refers to the representation of infrastructure resources,\r
+ namely: the physical resources, the virtualization\r
+ facility resources and the virtual resources.\r
+\r
+Physical Resource\r
+ The term refers to a hardware pieces of the NFV infrastructure, which may\r
+ also include the firmware which enables the hardware.\r
+\r
+Virtual Resource\r
+ The term refers to a resource, which is provided as services built on top\r
+ of the physical resources via the virtualization facilities; in particular,\r
+ they are the resources on which VNF entities are deployed, e.g.\r
+ the VMs, virtual switches, virtual routers, virtual disks etc.\r
+\r
+Visualization Facility\r
+ The term refers to a resource that enables the creation\r
+ of virtual environments on top of the physical resources, e.g.\r
+ hypervisor, OpenStack, etc.\r
+\r
+Upgrade Campaign\r
+ The term refers to a choreography that describes how the upgrade should\r
+ be performed in terms of its targets (i.e. upgrade objects), the\r
+ steps/actions required of upgrading each, and the coordination of these\r
+ steps so that service availability can be maintained. It is an input to an\r
+ upgrade tool (Escalator) to carry out the upgrade.\r
+\r
+Upgrade Duration\r
+ The duration of an upgrade characterized by the time elapsed between its\r
+ initiation and its completion. E.g. from the moment the execution of an\r
+ upgrade campaign has started until it has been committed. Depending on\r
+ the upgrade method and its target some parts of the system may be in a more\r
+ vulnerable state.\r
+\r
+Outage\r
+ The period of time during which a given service is not provided is referred\r
+ as the outage of that given service. If a subsystem or the entire system\r
+ does not provide any service, it is the outage of the given subsystem or the\r
+ system. Smooth upgrade means upgrade with no outage for the user plane, i.e.\r
+ no VNF should experience service outage.\r
+\r
+Rollback\r
+ The term refers to a failure handling strategy that reverts the changes\r
+ done by a potentially failed upgrade execution one by one in a reverse order.\r
+ I.e. it is like undoing the changes done by the upgrade.\r
+\r
+Restore\r
+ The term refers to a failure handling strategy that reverts the changes\r
+ done by an upgrade by restoring the system from some backup data. This\r
+ results in the loss of any data persisted since the backup has been taken.\r
+\r
+Rollforward\r
+ The term refers to a failure handling strategy applied after a restore\r
+ (from a backup) opertaion to recover any loss of data persisted between\r
+ the time the backup has been taken and the moment it is restored. Rollforward\r
+ requires that data that needs to survive the restore operation is logged at\r
+ a location not impacted by the restore so that it can be re-applied to the\r
+ system after its restoration from the backup.\r
+\r
+Downgrade\r
+ The term refers to an upgrade in which an earlier version of the software\r
+ is restored through the upgrade procedure. A system can be downgraded to any\r
+ earlier version and the compatibility of the versions will determine the\r
+ applicable upgrade strategies and whether service outage can be avoided.\r
+ In particular any data conversion needs special attention.\r
+\r
+\r
+\r
+Upgrade Objects\r
+~~~~~~~~~~~~~~~\r
+\r
+Physical Resource\r
+^^^^^^^^^^^^^^^^^\r
+\r
+Most cloud infrastructures support the dynamic addition/removal of\r
+hardware. Accordingly a hardware upgrade could be done by adding the new\r
+piece of hardware and removing the old one. From the persepctive of smooth\r
+upgrade the orchestration/scheduling of this actions is the primary concern.\r
+Upgrading a physical resource may involve as well the upgrade of its firmware\r
+and/or modifying its configuration data. This may require the restart of the\r
+hardware.\r
+\r
+\r
+\r
+Virtual Resources\r
+^^^^^^^^^^^^^^^^^\r
+\r
+Addition and removal of virtual resources may be initiated by the users or be\r
+a result of an elasticity action. Users may also request the upgrade of their\r
+virtual resources using a new VM image.\r
+\r
+.. Needs to be moved to requirement section: Escalator should facilitate such an\r
+option and allow for a smooth upgrade.\r
+\r
+On the other hand changes in the infrastructure, namely, in the hardware and/or\r
+the virtualization facility resources may result in the upgrade of the virtual\r
+resources. For example if by some reason the hypervisor is changed and\r
+the current VMs cannot be migrated to the new hypervisor - they are\r
+incompatible - then the VMs need to be upgraded too. This is not\r
+something the NFVI user (i.e. VNFs ) would know about. In such cases\r
+smooth upgrade is essential.\r
+\r
+\r
+Virtualization Facility Resources\r
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r
+\r
+Based on the functionality they provide, virtualization facility\r
+resources could be divided into computing node, networking node,\r
+storage node and management node.\r
+\r
+The possible upgrade objects in these nodes are addressed below:\r
+(Note: hardware based virtualization may be considered as virtualization\r
+facility resource, but from escalator perspective, it is better to\r
+consider it as part of the hardware upgrade. )\r
+\r
+**Computing node**\r
+\r
+1. OS Kernel\r
+\r
+2. Hypvervisor and virtual switch\r
+\r
+3. Other kernel modules, like driver\r
+\r
+4. User space software packages, like nova-compute agents and other\r
+ control plane programs.\r
+\r
+Updating 1 and 2 will cause the loss of virtualzation functionality of\r
+the compute node, which may lead to data plane services interruption\r
+if the virtual resource is not redudant.\r
+\r
+Updating 3 might result the same.\r
+\r
+Updating 4 might lead to control plane services interruption if not an\r
+HA deployment.\r
+\r
+**Networking node**\r
+\r
+1. OS kernel, optional, not all switches/routers allow the upgrade their\r
+ OS since it is more like a firmware than a generic OS.\r
+\r
+2. User space software package, like neutron agents and other control\r
+ plane programs\r
+\r
+Updating 1 if allowed will cause a node reboot and therefore leads to\r
+data plane service interruption if the virtual resource is not\r
+redundant.\r
+\r
+Updating 2 might lead to control plane services interruption if not an\r
+HA deployment.\r
+\r
+**Storage node**\r
+\r
+1. OS kernel, optional, not all storage nodes allow the upgrade their OS\r
+ since it is more like a firmware than a generic OS.\r
+\r
+2. Kernel modules\r
+\r
+3. User space software packages, control plane programs\r
+\r
+Updating 1 if allowed will cause a node reboot and therefore leads to\r
+data plane services interruption if the virtual resource is not\r
+redundant.\r
+\r
+Update 2 might result in the same.\r
+\r
+Updating 3 might lead to control plane services interruption if not an\r
+HA deployment.\r
+\r
+**Management node**\r
+\r
+1. OS Kernel\r
+\r
+2. Kernel modules, like driver\r
+\r
+3. User space software packages, like database, message queue and\r
+ control plane programs.\r
+\r
+Updating 1 will cause a node reboot and therefore leads to control\r
+plane services interruption if not an HA deployment. Updating 2 might\r
+result in the same.\r
+\r
+Updating 3 might lead to control plane services interruption if not an\r
+HA deployment.\r
+\r
+\r
+\r
+\r
+\r
+Upgrade Granularity\r
+~~~~~~~~~~~~~~~~~~~\r
+\r
+The granularity of an upgrade can be characterized from two perspective:\r
+- the physical dimension and\r
+- the software dimension\r
+\r
+\r
+Physical Dimension\r
+^^^^^^^^^^^^^^^^^^\r
+\r
+The physical dimension characterizes the number of similar upgrade objects\r
+targeted by the upgrade, i.e. whether it is full / partial upgrade of a\r
+data centre, cluster, zone.\r
+Because of the upgrade of a data centre or a zone, it may be divided into\r
+several batches. Thus there is a need for efficiency in the execution of\r
+upgrades of potentially huge number of upgrade objects while still maintain\r
+availability to fulfill the requirement of smooth upgrade.\r
+\r
+The upgrade of a cloud environment (cluster) may also\r
+be partial. For example, in one cloud environment running a number of\r
+VNFs, we may just try to upgrade one of them to check the stability and\r
+performance, before we upgrade all of them.\r
+Thus there is a need for proper organization of the artifacts associated with\r
+the different upgrade objects. Also the different versions should be able\r
+to coextist beyond the upgrade period.\r
+\r
+From this perspective special attention may be needed when upgrading\r
+objects that are collaborating in a redundancy schema as in this case\r
+different versions not only need to coexist but also collaborate. This\r
+puts requirement on the upgrade objects primarily. If this is not possible\r
+the upgrade campaign should be designed in such a way that the proper\r
+isolation is ensured.\r
+\r
+Software Dimension\r
+^^^^^^^^^^^^^^^^^^\r
+\r
+The software dimension of the upgrade characterizes the upgrade object\r
+type targeted and the combination in which they are upgraded together.\r
+\r
+Even though the upgrade may\r
+initially target only one type of upgrade object, e.g. the hypervisor\r
+the dependency of other upgrade objects on this initial target object may\r
+require their upgrade as well. I.e. the upgrades need to be combined. From this\r
+perspective the main concern is compatibility of the dependent and\r
+sponsor objects. To take into consideration of these dependencies\r
+they need to be described together with the version compatility information.\r
+Breaking dependencies is the major cause of outages during upgrades.\r
+\r
+In other cases it is more efficient to upgrade a combination of upgrade\r
+objects than to do it one by one. One aspect of the combination is how\r
+the upgrade packages can be combined, whether a new image can be created for\r
+them before hand or the different packages can be installed during the upgrade\r
+independently, but activated together.\r
+\r
+The combination of upgrade objects may span across\r
+layers (e.g. software stack in the host and the VM of the VNF).\r
+Thus, it may require additional coordination between the management layers.\r
+\r
+With respect to each upgrade object type and even stacks we can\r
+distingush major and minor upgrades:\r
+\r
+**Major Upgrade**\r
+\r
+Upgrades between major releases may introducing significant changes in\r
+function, configuration and data, such as the upgrade of OPNFV from\r
+Arno to Brahmaputra.\r
+\r
+**Minor Upgrade**\r
+\r
+Upgrades inside one major releases which would not leads to changing\r
+the structure of the platform and may not infect the schema of the\r
+system data.\r
+\r
+Scope of Impact\r
+~~~~~~~~~~~~~~~\r
+\r
+Considering availability and therefore smooth upgrade, one of the major\r
+concerns is the predictability and control of the outcome of the different\r
+upgrade operations. Ideally an upgrade can be performed without impacting any\r
+entity in the system, which means none of the operations change or potentially\r
+change the behaviour of any entity in the system in an uncotrolled manner.\r
+Accordingly the operations of such an upgrade can be performed any time while\r
+the system is running, while all the entities are online. No entity needs to be\r
+taken offline to avoid such adverse effects. Hence such upgrade operations\r
+are referred as online operations. The effects of the upgrade might be activated\r
+next time it is used, or may require a special activation action such as a\r
+restart. Note that the activation action provides more control and predictability.\r
+\r
+If an entity's behavior in the system may change due to the upgrade it may\r
+be better to take it offline for the time of the relevant upgrade operations.\r
+The main question is however considering the hosting relation of an upgrade\r
+object what hosted entities are impacted. Accordingly we can identify a scope\r
+which is impacted by taking the given upgrade object offline. The entities\r
+that are in the scope of impact may need to be taken offline or moved out of\r
+this scope i.e. migrated.\r
+\r
+If the impacted entity is in a different layer managed by another manager\r
+this may require coordination because taking out of service some\r
+infrastructure resources for the time of their upgrade which support virtual\r
+resources used by VNFs that should not experience outages. The hosted VNFs\r
+may or may not allow for the hot migration of their VMs. In case of migration\r
+the VMs placement policy should be considered.\r
+\r
+\r
+\r
+Upgrade duration\r
+~~~~~~~~~~~~~~~~\r
+\r
+As the OPNFV end-users are primarily Telecom operators, the network\r
+services provided by the VNFs deployed on the NFVI should meet the\r
+requirement of 'Carrier Grade'.::\r
+\r
+ In telecommunication, a "carrier grade" or"carrier class" refers to a\r
+ system, or a hardware or software component that is extremely reliable,\r
+ well tested and proven in its capabilities. Carrier grade systems are\r
+ tested and engineered to meet or exceed "five nines" high availability\r
+ standards, and provide very fast fault recovery through redundancy\r
+ (normally less than 50 milliseconds). [from wikipedia.org]\r
+\r
+"five nines" means working all the time in ONE YEAR except 5'15".\r
+\r
+::\r
+\r
+ We have learnt that a well prepared upgrade of OpenStack needs 10\r
+ minutes. The major time slot in the outage time is used spent on\r
+ synchronizing the database. [from ' Ten minutes OpenStack Upgrade? Done!\r
+ ' by Symantec]\r
+\r
+This 10 minutes of downtime of the OpenStack services however did not impact the\r
+users, i.e. the VMs running on the compute nodes. This was the outage of\r
+the control plane only. On the other hand with respect to the\r
+preparations this was a manually tailored upgrade specific to the\r
+particular deployment and the versions of each OpenStack service.\r
+\r
+The project targets to achieve a more generic methodology, which however\r
+requires that the upgrade objects fulfil certain requirements. Since\r
+this is only possible on the long run we target first the upgrade\r
+of the different VIM services from version to version.\r
+\r
+**Questions:**\r
+\r
+1. Can we manage to upgrade OPNFV in only 5 minutes?\r
+ \r
+.. <MT> The first question is whether we have the same carrier grade\r
+ requirement on the control plane as on the user plane. I.e. how\r
+ much control plane outage we can/willing to tolerate?\r
+ In the above case probably if the database is only half of the size\r
+ we can do the upgrade in 5 minutes, but is that good? It also means\r
+ that if the database is twice as much then the outage is 20\r
+ minutes.\r
+ For the user plane we should go for less as with two release yearly\r
+ that means 10 minutes outage per year.\r
+\r
+.. <Malla> 10 minutes outage per year to the users? Plus, if we take\r
+ control plane into the consideration, then total outage will be\r
+ more than 10 minute in whole network, right?\r
+\r
+.. <MT> The control plane outage does not have to cause outage to\r
+ the users, but it may of course depending on the size of the system\r
+ as it's more likely that there's a failure that needs to be handled\r
+ by the control plane.\r
+\r
+2. Is it acceptable for end users ? Such as a planed service\r
+ interruption will lasting more than ten minutes for software\r
+ upgrade.\r
+\r
+.. <MT> For user plane, no it's not acceptable in case of\r
+ carrier-grade. The 5' 15" downtime should include unplanned and\r
+ planned downtimes.\r
+ \r
+.. <Malla> I go agree with Maria, it is not acceptable.\r
+\r
+3. Will any VNFs still working well when VIM is down?\r
+\r
+.. <MT> In case of OpenStack it seems yes. .:)\r
+\r
+The maximum duration of an upgrade\r
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r
+\r
+The duration of an upgrade is related to and proportional with the\r
+scale and the complexity of the OPNFV platform as well as the\r
+granularity (in function and in space) of the upgrade.\r
+\r
+.. <Malla> Also, if is a partial upgrade like module upgrade, it depends\r
+ also on the OPNFV modules and their tight connection entities as well.\r
+\r
+.. <MT> Since the maintenance window is shrinking and becoming non-existent\r
+ the duration of the upgrade is secondary to the requirement of smooth upgrade.\r
+ But probably we want to be able to put a time constraint on each upgrade\r
+ during which it must complete otherwise it is considered failed and the system\r
+ should be rolled back. I.e. in case of automatic execution it might not be clear\r
+ if an upgrade is long or just hanging. The time constraints may be a function\r
+ of the size of the system in terms of the upgrade object(s).\r
+\r
+The maximum duration of a roll back when an upgrade is failed \r
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r
+\r
+The duration of a roll back is short than the corresponding upgrade. It\r
+depends on the duration of restore the software and configure data from\r
+pre-upgrade backup / snapshot.\r
+\r
+.. <MT> During the upgrade process two types of failure may happen:\r
+ In case we can recover from the failure by undoing the upgrade\r
+ actions it is possible to roll back the already executed part of the\r
+ upgrade in graceful manner introducing no more service outage than\r
+ what was introduced during the upgrade. Such a graceful roll back\r
+ requires typically the same amount of time as the executed portion of\r
+ the upgrade and impose minimal state/data loss.\r
+ \r
+.. <MT> Requirement: It should be possible to roll back gracefully the\r
+ failed upgrade of stateful services of the control plane.\r
+ In case we cannot recover from the failure by just undoing the\r
+ upgrade actions, we have to restore the upgraded entities from their\r
+ backed up state. In other terms the system falls back to an earlier\r
+ state, which is typically a faster recovery procedure than graceful\r
+ roll back and depending on the statefulness of the entities involved it\r
+ may result in significant state/data loss.\r
+ \r
+.. <MT> Two possible types of failures can happen during an upgrade\r
+\r
+.. <MT> We can recover from the failure that occurred in the upgrade process:\r
+ In this case, a graceful rolling back of the executed part of the\r
+ upgrade may be possible which would "undo" the executed part in a\r
+ similar fashion. Thus, such a roll back introduces no more service\r
+ outage during an upgrade than the executed part introduced. This\r
+ process typically requires the same amount of time as the executed\r
+ portion of the upgrade and impose minimal state/data loss.\r
+\r
+.. <MT> We cannot recover from the failure that occurred in the upgrade\r
+ process: In this case, the system needs to fall back to an earlier\r
+ consistent state by reloading this backed-up state. This is typically\r
+ a faster recovery procedure than the graceful roll back, but can cause\r
+ state/data loss. The state/data loss usually depends on the\r
+ statefulness of the entities whose state is restored from the backup.\r
+\r
+The maximum duration of a VNF interruption (Service outage)\r
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r
+\r
+Since not the entire process of a smooth upgrade will affect the VNFs,\r
+the duration of the VNF interruption may be shorter than the duration\r
+of the upgrade. In some cases, the VNF running without the control\r
+from of the VIM is acceptable.\r
+\r
+.. <MT> Should require explicitly that the NFVI should be able to\r
+ provide its services to the VNFs independent of the control plane?\r
+\r
+.. <MT> Requirement: The upgrade of the control plane must not cause\r
+ interruption of the NFVI services provided to the VNFs.\r
+\r
+.. <MT> With respect to carrier-grade the yearly service outage of the\r
+ VNF should not exceed 5' 15" regardless whether it is planned or\r
+ unplanned outage. Considering the HA requirements TL-9000 requires an\r
+ end-to-end service recovery time of 15 seconds based on which the ETSI\r
+ GS NFV-REL 001 V1.1.1 (2015-01) document defines three service\r
+ availability levels (SAL). The proposed example service recovery times\r
+ for these levels are:\r
+\r
+.. <MT> SAL1: 5-6 seconds\r
+\r
+.. <MT> SAL2: 10-15 seconds\r
+\r
+.. <MT> SAL3: 20-25 seconds\r
+\r
+.. <Pva> my comment was actually that the downtime metrics of the\r
+ underlying elements, components and services are small fraction of the\r
+ total E2E service availability time. No-one on the E2E service path\r
+ will get the whole downtime allocation (in this context it includes\r
+ upgrade process related outages for the services provided by VIM etc.\r
+ elements that are subject to upgrade process).\r
+ \r
+.. <MT> So what you are saying is that the upgrade of any entity\r
+ (component, service) shouldn't cause even this much service\r
+ interruption. This was the reason I brought these figures here as well\r
+ that they are posing some kind of upper-upper boundary. Ideally the\r
+ interruption is in the millisecond range i.e. no more than a\r
+ switch-over or a live migration.\r
+ \r
+.. <MT> Requirement: Any interruption caused to the VNF by the upgrade\r
+ of the NFVI should be in the sub-second range.\r
+\r
+.. <MT]> In the future we also need to consider the upgrade of the NFVI,\r
+ i.e. HW, firmware, hypervisors, host OS etc.
\ No newline at end of file
Code Review / escalator.git / blobdiff