1 General Requirements Background and Terminology
\r
2 -----------------------------------------------
\r
4 Terminologies and definitions
\r
5 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
\r
8 The term is an abbreviation for Network Function Virtualization
\r
9 Infrastructure; sometimes it is also referred as data plane in this
\r
10 document. The NFVI provides the virtual resources to the virtual
\r
11 network functions under the control of the VIM.
\r
14 The term is an abbreviation for Virtual Infrastructure Manager;
\r
15 sometimes it is also referred as control plane in this document.
\r
16 The VIM controls and manages the NFVI compute, network and storage
\r
17 resources to provide the required virtual resources to the VNFs.
\r
20 The term refers to network service providers and Virtual Network
\r
21 Function (VNF) providers.
\r
24 The term refers to a subscriber of the Operator's services.
\r
27 The term refers to a service provided by an Operator to its
\r
28 end-users using a set of (virtualized) Network Functions
\r
30 Infrastructure Services
\r
31 The term refers to services provided by the NFV Infrastructure to the VNFs
\r
32 as required by the Management & Orchestration functions and especially the VIM.
\r
33 I.e. these are the virtual resources as perceived by the VNFs.
\r
36 The term refers to an upgrade that results in no service outage
\r
40 The term refers to an upgrade strategy, which upgrades a node or a subset
\r
41 of nodes at a time in a wave style rolling through the data centre. It
\r
42 is a popular upgrade strategy to maintain service availability.
\r
44 Parallel Universe Upgrade
\r
45 The term refers to an upgrade strategy, which creates and deploys
\r
46 a new universe - a system with the new configuration - while the old
\r
47 system continues running. The state of the old system is transferred
\r
48 to the new system after sufficient testing of the new system.
\r
50 Infrastructure Resource Model
\r
51 The term refers to the representation of infrastructure resources,
\r
52 namely: the physical resources, the virtualization
\r
53 facility resources and the virtual resources.
\r
56 The term refers to a piece of hardware in the NFV infrastructure that may
\r
57 also include firmware enabling this piece of hardware.
\r
60 The term refers to a resource, which is provided as services built on top
\r
61 of the physical resources via the virtualization facilities; in particular,
\r
62 virtual resources are the resources on which VNFs are deployed. Examples of
\r
63 virtual resources are: VMs, virtual switches, virtual routers, virtual disks.
\r
65 Visualization Facility
\r
66 The term refers to a resource that enables the creation
\r
67 of virtual environments on top of the physical resources, e.g.
\r
68 hypervisor, OpenStack, etc.
\r
71 The term refers to a choreography that describes how the upgrade should
\r
72 be performed in terms of its targets (i.e. upgrade objects), the
\r
73 steps/actions required of upgrading each, and the coordination of these
\r
74 steps so that service availability can be maintained. It is an input to an
\r
75 upgrade tool (Escalator) to carry out the upgrade.
\r
78 The duration of an upgrade characterized by the time elapsed between its
\r
79 initiation and its completion. E.g. from the moment the execution of an
\r
80 upgrade campaign has started until it has been committed. Depending on
\r
81 the upgrade strategy, the state of the configuration and the upgrade target
\r
82 some parts of the system may be in a more vulnerable state with respect to
\r
83 service availbility.
\r
86 The period of time during which a given service is not provided is referred
\r
87 as the outage of that given service. If a subsystem or the entire system
\r
88 does not provide any service, it is the outage of the given subsystem or the
\r
89 system. Smooth upgrade means upgrade with no outage for the user plane, i.e.
\r
90 no VNF should experience service outage.
\r
93 The term refers to a failure handling strategy that reverts the changes
\r
94 done by a potentially failed upgrade execution one by one in a reverse order.
\r
95 I.e. it is like undoing the changes done by the upgrade.
\r
98 The term refers to data persisted to a storage, so that it can be used to
\r
99 restore the system or a given part of it in the same state as it was when the
\r
100 backup was created assuming a cold restart. Changes made to the system from
\r
101 the moment the backup was created till the moment it is used to restore the
\r
102 (sub)system are lost in the restoration process.
\r
105 The term refers to a failure handling strategy that reverts the changes
\r
106 done, for example, by an upgrade by restoring the system from some backup
\r
107 data. This results in the loss of any change and data persisted after the
\r
108 backup was been taken. To recover those additional measures need to be taken
\r
109 if necessary (e.g. rollforward).
\r
112 The term refers to a failure handling strategy applied after a restore
\r
113 (from a backup) opertaion to recover any loss of data persisted between
\r
114 the time the backup has been taken and the moment it is restored. Rollforward
\r
115 requires that data that needs to survive the restore operation is logged at
\r
116 a location not impacted by the restore so that it can be re-applied to the
\r
117 system after its restoration from the backup.
\r
120 The term refers to an upgrade in which an earlier version of the software
\r
121 is restored through the upgrade procedure. A system can be downgraded to any
\r
122 earlier version and the compatibility of the versions will determine the
\r
123 applicable upgrade strategies and whether service outage can be avoided.
\r
124 In particular any data conversion needs special attention.
\r
134 Most cloud infrastructures support the dynamic addition and removal of
\r
135 hardware. Accordingly a hardware upgrade could be done by adding the new
\r
136 piece of hardware and removing the old one. From the persepctive of smooth
\r
137 upgrade the orchestration/scheduling of these actions is the primary concern.
\r
139 Upgrading a physical resource may involve as well the upgrade of its firmware
\r
140 and/or modifying its configuration data. This may require the restart of the
\r
148 Addition and removal of virtual resources may be initiated by the users or be
\r
149 a result of an elasticity action. Users may also request the upgrade of their
\r
150 virtual resources using a new VM image.
\r
152 .. Needs to be moved to requirement section: Escalator should facilitate such an
\r
153 option and allow for a smooth upgrade.
\r
155 On the other hand changes in the infrastructure, namely, in the hardware and/or
\r
156 the virtualization facility resources may result in the upgrade of the virtual
\r
157 resources. For example if by some reason the hypervisor is changed and
\r
158 the current VMs cannot be migrated to the new hypervisor - they are
\r
159 incompatible - then the VMs need to be upgraded too. This is not
\r
160 something the NFVI user (i.e. VNFs ) would know about.
\r
163 Virtualization Facility Resources
\r
164 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
\r
166 Based on the functionality they provide, virtualization facility
\r
167 resources could be divided into computing node, networking node,
\r
168 storage node and management node.
\r
170 The possible upgrade objects in these nodes are considered below:
\r
171 (Note: hardware based virtualization may be considered as virtualization
\r
172 facility resource, but from escalator perspective, it is better to
\r
173 consider it as part of the hardware upgrade. )
\r
179 2. Hypvervisor and virtual switch
\r
181 3. Other kernel modules, like drivers
\r
183 4. User space software packages, like nova-compute agents and other
\r
184 control plane programs.
\r
186 Updating 1 and 2 will cause the loss of virtualzation functionality of
\r
187 the compute node, which may lead to the interruption of data plane services
\r
188 if the virtual resource is not redudant.
\r
190 Updating 3 might have the same result.
\r
192 Updating 4 might lead to control plane services interruption if not an
\r
195 .. <MT> I'm not sure why would 4 cause control plane interruption on a
\r
196 compute node. My understanding is that simply the node cannot be managed.
\r
197 Redundancy won't help in that either.
\r
200 **Networking node**
\r
202 1. OS kernel, optional, not all switches/routers allow the upgrade their
\r
203 OS since it is more like a firmware than a generic OS.
\r
205 2. User space software package, like neutron agents and other control
\r
208 Updating 1 if allowed will cause a node reboot and therefore leads to
\r
209 data plane service interruption if the virtual resource is not
\r
212 Updating 2 might lead to control plane services interruption if not an
\r
217 1. OS kernel, optional, not all storage nodes allow the upgrade their OS
\r
218 since it is more like a firmware than a generic OS.
\r
222 3. User space software packages, control plane programs
\r
224 Updating 1 if allowed will cause a node reboot and therefore leads to
\r
225 data plane services interruption if the virtual resource is not
\r
228 Update 2 might result in the same.
\r
230 Updating 3 might lead to control plane services interruption if not an
\r
233 **Management node**
\r
237 2. Kernel modules, like driver
\r
239 3. User space software packages, like database, message queue and
\r
240 control plane programs.
\r
242 Updating 1 will cause a node reboot and therefore leads to control
\r
243 plane services interruption if not an HA deployment. Updating 2 might
\r
244 result in the same.
\r
246 Updating 3 might lead to control plane services interruption if not an
\r
253 Upgrade Granularity
\r
254 ~~~~~~~~~~~~~~~~~~~
\r
256 The granularity of an upgrade can be characterized from two perspective:
\r
257 - the physical dimension and
\r
258 - the software dimension
\r
264 The physical dimension characterizes the number of similar upgrade objects
\r
265 targeted by the upgrade, i.e. whether it is full / partial upgrade of a
\r
266 data centre, cluster, zone.
\r
267 Because of the upgrade of a data centre or a zone, it may be divided into
\r
268 several batches. Thus there is a need for efficiency in the execution of
\r
269 upgrades of potentially huge number of upgrade objects while still maintain
\r
270 availability to fulfill the requirement of smooth upgrade.
\r
272 The upgrade of a cloud environment (cluster) may also
\r
273 be partial. For example, in one cloud environment running a number of
\r
274 VNFs, we may just try to upgrade one of them to check the stability and
\r
275 performance, before we upgrade all of them.
\r
276 Thus there is a need for proper organization of the artifacts associated with
\r
277 the different upgrade objects. Also the different versions should be able
\r
278 to coextist beyond the upgrade period.
\r
280 From this perspective special attention may be needed when upgrading
\r
281 objects that are collaborating in a redundancy schema as in this case
\r
282 different versions not only need to coexist but also collaborate. This
\r
283 puts requirement on the upgrade objects primarily. If this is not possible
\r
284 the upgrade campaign should be designed in such a way that the proper
\r
285 isolation is ensured.
\r
290 The software dimension of the upgrade characterizes the upgrade object
\r
291 type targeted and the combination in which they are upgraded together.
\r
293 Even though the upgrade may
\r
294 initially target only one type of upgrade object, e.g. the hypervisor
\r
295 the dependency of other upgrade objects on this initial target object may
\r
296 require their upgrade as well. I.e. the upgrades need to be combined. From this
\r
297 perspective the main concern is compatibility of the dependent and
\r
298 sponsor objects. To take into consideration of these dependencies
\r
299 they need to be described together with the version compatility information.
\r
300 Breaking dependencies is the major cause of outages during upgrades.
\r
302 In other cases it is more efficient to upgrade a combination of upgrade
\r
303 objects than to do it one by one. One aspect of the combination is how
\r
304 the upgrade packages can be combined, whether a new image can be created for
\r
305 them before hand or the different packages can be installed during the upgrade
\r
306 independently, but activated together.
\r
308 The combination of upgrade objects may span across
\r
309 layers (e.g. software stack in the host and the VM of the VNF).
\r
310 Thus, it may require additional coordination between the management layers.
\r
312 With respect to each upgrade object type and even stacks we can
\r
313 distingush major and minor upgrades:
\r
317 Upgrades between major releases may introducing significant changes in
\r
318 function, configuration and data, such as the upgrade of OPNFV from
\r
319 Arno to Brahmaputra.
\r
323 Upgrades inside one major releases which would not leads to changing
\r
324 the structure of the platform and may not infect the schema of the
\r
330 Considering availability and therefore smooth upgrade, one of the major
\r
331 concerns is the predictability and control of the outcome of the different
\r
332 upgrade operations. Ideally an upgrade can be performed without impacting any
\r
333 entity in the system, which means none of the operations change or potentially
\r
334 change the behaviour of any entity in the system in an uncotrolled manner.
\r
335 Accordingly the operations of such an upgrade can be performed any time while
\r
336 the system is running, while all the entities are online. No entity needs to be
\r
337 taken offline to avoid such adverse effects. Hence such upgrade operations
\r
338 are referred as online operations. The effects of the upgrade might be activated
\r
339 next time it is used, or may require a special activation action such as a
\r
340 restart. Note that the activation action provides more control and predictability.
\r
342 If an entity's behavior in the system may change due to the upgrade it may
\r
343 be better to take it offline for the time of the relevant upgrade operations.
\r
344 The main question is however considering the hosting relation of an upgrade
\r
345 object what hosted entities are impacted. Accordingly we can identify a scope
\r
346 which is impacted by taking the given upgrade object offline. The entities
\r
347 that are in the scope of impact may need to be taken offline or moved out of
\r
348 this scope i.e. migrated.
\r
350 If the impacted entity is in a different layer managed by another manager
\r
351 this may require coordination because taking out of service some
\r
352 infrastructure resources for the time of their upgrade which support virtual
\r
353 resources used by VNFs that should not experience outages. The hosted VNFs
\r
354 may or may not allow for the hot migration of their VMs. In case of migration
\r
355 the VMs placement policy should be considered.
\r
362 As the OPNFV end-users are primarily Telecom operators, the network
\r
363 services provided by the VNFs deployed on the NFVI should meet the
\r
364 requirement of 'Carrier Grade'.::
\r
366 In telecommunication, a "carrier grade" or"carrier class" refers to a
\r
367 system, or a hardware or software component that is extremely reliable,
\r
368 well tested and proven in its capabilities. Carrier grade systems are
\r
369 tested and engineered to meet or exceed "five nines" high availability
\r
370 standards, and provide very fast fault recovery through redundancy
\r
371 (normally less than 50 milliseconds). [from wikipedia.org]
\r
373 "five nines" means working all the time in ONE YEAR except 5'15".
\r
377 We have learnt that a well prepared upgrade of OpenStack needs 10
\r
378 minutes. The major time slot in the outage time is used spent on
\r
379 synchronizing the database. [from ' Ten minutes OpenStack Upgrade? Done!
\r
382 This 10 minutes of downtime of the OpenStack services however did not impact the
\r
383 users, i.e. the VMs running on the compute nodes. This was the outage of
\r
384 the control plane only. On the other hand with respect to the
\r
385 preparations this was a manually tailored upgrade specific to the
\r
386 particular deployment and the versions of each OpenStack service.
\r
388 The project targets to achieve a more generic methodology, which however
\r
389 requires that the upgrade objects fulfil certain requirements. Since
\r
390 this is only possible on the long run we target first the upgrade
\r
391 of the different VIM services from version to version.
\r
395 1. Can we manage to upgrade OPNFV in only 5 minutes?
\r
397 .. <MT> The first question is whether we have the same carrier grade
\r
398 requirement on the control plane as on the user plane. I.e. how
\r
399 much control plane outage we can/willing to tolerate?
\r
400 In the above case probably if the database is only half of the size
\r
401 we can do the upgrade in 5 minutes, but is that good? It also means
\r
402 that if the database is twice as much then the outage is 20
\r
404 For the user plane we should go for less as with two release yearly
\r
405 that means 10 minutes outage per year.
\r
407 .. <Malla> 10 minutes outage per year to the users? Plus, if we take
\r
408 control plane into the consideration, then total outage will be
\r
409 more than 10 minute in whole network, right?
\r
411 .. <MT> The control plane outage does not have to cause outage to
\r
412 the users, but it may of course depending on the size of the system
\r
413 as it's more likely that there's a failure that needs to be handled
\r
414 by the control plane.
\r
416 2. Is it acceptable for end users ? Such as a planed service
\r
417 interruption will lasting more than ten minutes for software
\r
420 .. <MT> For user plane, no it's not acceptable in case of
\r
421 carrier-grade. The 5' 15" downtime should include unplanned and
\r
424 .. <Malla> I go agree with Maria, it is not acceptable.
\r
426 3. Will any VNFs still working well when VIM is down?
\r
428 .. <MT> In case of OpenStack it seems yes. .:)
\r
430 The maximum duration of an upgrade
\r
431 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
\r
433 The duration of an upgrade is related to and proportional with the
\r
434 scale and the complexity of the OPNFV platform as well as the
\r
435 granularity (in function and in space) of the upgrade.
\r
437 .. <Malla> Also, if is a partial upgrade like module upgrade, it depends
\r
438 also on the OPNFV modules and their tight connection entities as well.
\r
440 .. <MT> Since the maintenance window is shrinking and becoming non-existent
\r
441 the duration of the upgrade is secondary to the requirement of smooth upgrade.
\r
442 But probably we want to be able to put a time constraint on each upgrade
\r
443 during which it must complete otherwise it is considered failed and the system
\r
444 should be rolled back. I.e. in case of automatic execution it might not be clear
\r
445 if an upgrade is long or just hanging. The time constraints may be a function
\r
446 of the size of the system in terms of the upgrade object(s).
\r
448 The maximum duration of a roll back when an upgrade is failed
\r
449 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
\r
451 The duration of a roll back is short than the corresponding upgrade. It
\r
452 depends on the duration of restore the software and configure data from
\r
453 pre-upgrade backup / snapshot.
\r
455 .. <MT> During the upgrade process two types of failure may happen:
\r
456 In case we can recover from the failure by undoing the upgrade
\r
457 actions it is possible to roll back the already executed part of the
\r
458 upgrade in graceful manner introducing no more service outage than
\r
459 what was introduced during the upgrade. Such a graceful roll back
\r
460 requires typically the same amount of time as the executed portion of
\r
461 the upgrade and impose minimal state/data loss.
\r
463 .. <MT> Requirement: It should be possible to roll back gracefully the
\r
464 failed upgrade of stateful services of the control plane.
\r
465 In case we cannot recover from the failure by just undoing the
\r
466 upgrade actions, we have to restore the upgraded entities from their
\r
467 backed up state. In other terms the system falls back to an earlier
\r
468 state, which is typically a faster recovery procedure than graceful
\r
469 roll back and depending on the statefulness of the entities involved it
\r
470 may result in significant state/data loss.
\r
472 .. <MT> Two possible types of failures can happen during an upgrade
\r
474 .. <MT> We can recover from the failure that occurred in the upgrade process:
\r
475 In this case, a graceful rolling back of the executed part of the
\r
476 upgrade may be possible which would "undo" the executed part in a
\r
477 similar fashion. Thus, such a roll back introduces no more service
\r
478 outage during an upgrade than the executed part introduced. This
\r
479 process typically requires the same amount of time as the executed
\r
480 portion of the upgrade and impose minimal state/data loss.
\r
482 .. <MT> We cannot recover from the failure that occurred in the upgrade
\r
483 process: In this case, the system needs to fall back to an earlier
\r
484 consistent state by reloading this backed-up state. This is typically
\r
485 a faster recovery procedure than the graceful roll back, but can cause
\r
486 state/data loss. The state/data loss usually depends on the
\r
487 statefulness of the entities whose state is restored from the backup.
\r
489 The maximum duration of a VNF interruption (Service outage)
\r
490 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
\r
492 Since not the entire process of a smooth upgrade will affect the VNFs,
\r
493 the duration of the VNF interruption may be shorter than the duration
\r
494 of the upgrade. In some cases, the VNF running without the control
\r
495 from of the VIM is acceptable.
\r
497 .. <MT> Should require explicitly that the NFVI should be able to
\r
498 provide its services to the VNFs independent of the control plane?
\r
500 .. <MT> Requirement: The upgrade of the control plane must not cause
\r
501 interruption of the NFVI services provided to the VNFs.
\r
503 .. <MT> With respect to carrier-grade the yearly service outage of the
\r
504 VNF should not exceed 5' 15" regardless whether it is planned or
\r
505 unplanned outage. Considering the HA requirements TL-9000 requires an
\r
506 end-to-end service recovery time of 15 seconds based on which the ETSI
\r
507 GS NFV-REL 001 V1.1.1 (2015-01) document defines three service
\r
508 availability levels (SAL). The proposed example service recovery times
\r
509 for these levels are:
\r
511 .. <MT> SAL1: 5-6 seconds
\r
513 .. <MT> SAL2: 10-15 seconds
\r
515 .. <MT> SAL3: 20-25 seconds
\r
517 .. <Pva> my comment was actually that the downtime metrics of the
\r
518 underlying elements, components and services are small fraction of the
\r
519 total E2E service availability time. No-one on the E2E service path
\r
520 will get the whole downtime allocation (in this context it includes
\r
521 upgrade process related outages for the services provided by VIM etc.
\r
522 elements that are subject to upgrade process).
\r
524 .. <MT> So what you are saying is that the upgrade of any entity
\r
525 (component, service) shouldn't cause even this much service
\r
526 interruption. This was the reason I brought these figures here as well
\r
527 that they are posing some kind of upper-upper boundary. Ideally the
\r
528 interruption is in the millisecond range i.e. no more than a
\r
529 switch-over or a live migration.
\r
531 .. <MT> Requirement: Any interruption caused to the VNF by the upgrade
\r
532 of the NFVI should be in the sub-second range.
\r
534 .. <MT]> In the future we also need to consider the upgrade of the NFVI,
\r
535 i.e. HW, firmware, hypervisors, host OS etc.