1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
4 High level architecture and general features
5 ============================================
10 The Doctor project circles around two distinct use cases: 1) management of
11 failures of virtualized resources and 2) planned maintenance, e.g. migration, of
12 virtualized resources. Both of them may affect a VNF/application and the network
13 service it provides, but there is a difference in frequency and how they can be
16 Failures are spontaneous events that may or may not have an impact on the
17 virtual resources. The Consumer should as soon as possible react to the failure,
18 e.g., by switching to the STBY node. The Consumer will then instruct the VIM on
19 how to clean up or repair the lost virtual resources, i.e. restore the VM, VLAN
20 or virtualized storage. How much the applications are affected varies.
21 Applications with built-in HA support might experience a short decrease in
22 retainability (e.g. an ongoing session might be lost) while keeping availability
23 (establishment or re-establishment of sessions are not affected), whereas the
24 impact on applications without built-in HA may be more serious. How much the
25 network service is impacted depends on how the service is implemented. With
26 sufficient network redundancy the service may be unaffected even when a specific
29 On the other hand, planned maintenance impacting virtualized resources are events
30 that are known in advance. This group includes e.g. migration due to software
31 upgrades of OS and hypervisor on a compute host. Some of these might have been
32 requested by the application or its management solution, but there is also a
33 need for coordination on the actual operations on the virtual resources. There
34 may be an impact on the applications and the service, but since they are not
35 spontaneous events there is room for planning and coordination between the
36 application management organization and the infrastructure management
37 organization, including performing whatever actions that would be required to
38 minimize the problems.
40 Failure prediction is the process of pro-actively identifying situations that
41 may lead to a failure in the future unless acted on by means of maintenance
42 activities. From applications' point of view, failure prediction may impact them
43 in two ways: either the warning time is so short that the application or its
44 management solution does not have time to react, in which case it is equal to
45 the failure scenario, or there is sufficient time to avoid the consequences by
46 means of maintenance activities, in which case it is similar to planned
52 NFV and the Cloud platform provide virtual resources and related control
53 functionality to users and administrators. :numref:`figure3` shows the high
54 level architecture of NFV focusing on the NFVI, i.e., the virtualized
55 infrastructure. The NFVI provides virtual resources, such as virtual machines
56 (VM) and virtual networks. Those virtual resources are used to run applications,
57 i.e. VNFs, which could be components of a network service which is managed by
58 the consumer of the NFVI. The VIM provides functionalities of controlling and
59 viewing virtual resources on hardware (physical) resources to the consumers,
60 i.e., users and administrators. OpenStack is a prominent candidate for this VIM.
61 The administrator may also directly control the NFVI without using the VIM.
63 Although OpenStack is the target upstream project where the new functional
64 elements (Controller, Notifier, Monitor, and Inspector) are expected to be
65 implemented, a particular implementation method is not assumed. Some of these
66 elements may sit outside of OpenStack and offer a northbound interface to
69 General Features and Requirements
70 ---------------------------------
72 The following features are required for the VIM to achieve high availability of
73 applications (e.g., MME, S/P-GW) and the Network Services:
75 1. Monitoring: Monitor physical and virtual resources.
76 2. Detection: Detect unavailability of physical resources.
77 3. Correlation and Cognition: Correlate faults and identify affected virtual
79 4. Notification: Notify unavailable virtual resources to their Consumer(s).
80 5. Fencing: Shut down or isolate a faulty resource.
81 6. Recovery action: Execute actions to process fault recovery and maintenance.
83 The time interval between the instant that an event is detected by the
84 monitoring system and the Consumer notification of unavailable resources shall
85 be < 1 second (e.g., Step 1 to Step 4 in :numref:`figure4`).
87 .. figure:: images/figure3.png
91 High level architecture
96 The VIM shall monitor physical and virtual resources for unavailability and
102 The VIM shall detect unavailability and failures of physical resources that
103 might cause errors/faults in virtual resources running on top of them.
104 Unavailability of physical resource is detected by various monitoring and
105 managing tools for hardware and software components. This may include also
106 predicting upcoming faults. Note, fault prediction is out of scope of this
107 project and is investigated in the OPNFV "Data Collection for Failure
108 Prediction" project [PRED]_.
110 The fault items/events to be detected shall be configurable.
112 The configuration shall enable Failure Selection and Aggregation. Failure
113 aggregation means the VIM determines unavailability of physical resource from
114 more than two non-critical failures related to the same resource.
116 There are two types of unavailability - immediate and future:
118 * Immediate unavailability can be detected by setting traps of raw failures on
119 hardware monitoring tools.
120 * Future unavailability can be found by receiving maintenance instructions
121 issued by the administrator of the NFVI or by failure prediction mechanisms.
123 Correlation and Cognition
124 ^^^^^^^^^^^^^^^^^^^^^^^^^
126 The VIM shall correlate each fault to the impacted virtual resource, i.e., the
127 VIM shall identify unavailability of virtualized resources that are or will be
128 affected by failures on the physical resources under them. Unavailability of a
129 virtualized resource is determined by referring to the mapping of physical and
130 virtualized resources.
132 VIM shall allow configuration of fault correlation between physical and
133 virtual resources. VIM shall support correlating faults:
135 * between a physical resource and another physical resource
136 * between a physical resource and a virtual resource
137 * between a virtual resource and another virtual resource
139 Failure aggregation is also required in this feature, e.g., a user may request
140 to be only notified if failures on more than two standby VMs in an (N+M)
141 deployment model occurred.
146 The VIM shall notify the alarm, i.e., unavailability of virtual resource(s), to
147 the Consumer owning it over the northbound interface, such that the Consumers
148 impacted by the failure can take appropriate actions to recover from the
151 The VIM shall also notify the unavailability of physical resources to its
154 All notifications shall be transferred immediately in order to minimize the
155 stalling time of the network service and to avoid over assignment caused by
156 delay of capability updates.
158 There may be multiple consumers, so the VIM has to find out the owner of a
159 faulty resource. Moreover, there may be a large number of virtual and physical
160 resources in a real deployment, so polling the state of all resources to the VIM
161 would lead to heavy signaling traffic. Thus, a publication/subscription
162 messaging model is better suited for these notifications, as notifications are
163 only sent to subscribed consumers.
165 Notifications will be send out along with the configuration by the consumer.
166 The configuration includes endpoint(s) in which the consumers can specify
167 multiple targets for the notification subscription, so that various and
168 multiple receiver functions can consume the notification message.
169 Also, the conditions for notifications shall be configurable, such that
170 the consumer can set according policies, e.g. whether it wants to receive
171 fault notifications or not.
173 Note: the VIM should only accept notification subscriptions for each resource
174 by its owner or administrator.
175 Notifications to the Consumer about the unavailability of virtualized
176 resources will include a description of the fault, preferably with sufficient
177 abstraction rather than detailed physical fault information.
183 Recovery actions, e.g. safe VM evacuation, have to be preceded by fencing the
184 failed host. Fencing hereby means to isolate or shut down a faulty resource.
185 Without fencing -- when the perceived disconnection is due to some transient
186 or partial failure -- the evacuation might lead into two identical instances
187 running together and having a dangerous conflict.
189 There is a cross-project definition in OpenStack of how to implement
190 fencing, but there has not been any progress. The general description is
192 https://wiki.openstack.org/wiki/Fencing_Instances_of_an_Unreachable_Host
194 OpenStack provides some mechanisms that allow fencing of faulty resources. Some
195 are automatically invoked by the platform itself (e.g. Nova disables the
196 compute service when libvirtd stops running, preventing new VMs to be scheduled
197 to that node), while other mechanisms are consumer trigger-based actions (e.g.
198 Neutron port admin-state-up). For other fencing actions not supported by
199 OpenStack, the Doctor project may suggest ways to address the gap (e.g. through
200 means of resourcing to external tools and orchestration methods), or
201 documenting or implementing them upstream.
203 The Doctor Inspector component will be responsible of marking resources down in
204 the OpenStack and back up if necessary.
209 In the basic :ref:`uc-fault1` use case, no automatic actions will be taken by
210 the VIM, but all recovery actions executed by the VIM and the NFVI will be
211 instructed and coordinated by the Consumer.
213 In a more advanced use case, the VIM may be able to recover the failed virtual
214 resources according to a pre-defined behavior for that resource. In principle
215 this means that the owner of the resource (i.e., its consumer or administrator)
216 can define which recovery actions shall be taken by the VIM. Examples are a
217 restart of the VM or migration/evacuation of the VM.
221 High level northbound interface specification
222 ---------------------------------------------
227 This interface allows the Consumer to subscribe to fault notification from the
228 VIM. Using a filter, the Consumer can narrow down which faults should be
229 notified. A fault notification may trigger the Consumer to switch from ACT to
230 STBY configuration and initiate fault recovery actions. A fault query
231 request/response message exchange allows the Consumer to find out about active
232 alarms at the VIM. A filter can be used to narrow down the alarms returned in
233 the response message.
235 .. figure:: images/figure4.png
239 High-level message flow for fault management
241 The high level message flow for the fault management use case is shown in
243 It consists of the following steps:
245 1. The VIM monitors the physical and virtual resources and the fault management
246 workflow is triggered by a monitored fault event.
247 2. Event correlation, fault detection and aggregation in VIM. Note: this may
248 also happen after Step 3.
249 3. Database lookup to find the virtual resources affected by the detected fault.
250 4. Fault notification to Consumer.
251 5. The Consumer switches to standby configuration (STBY).
252 6. Instructions to VIM requesting certain actions to be performed on the
253 affected resources, for example migrate/update/terminate specific
254 resource(s). After reception of such instructions, the VIM is executing the
255 requested action, e.g., it will migrate or terminate a virtual resource.
260 The NFVI maintenance interface allows the Administrator to notify the VIM about
261 a planned maintenance operation on the NFVI. A maintenance operation may for
262 example be an update of the server firmware or the hypervisor. The
263 MaintenanceRequest message contains instructions to change the state of the
264 physical resource from 'enabled' to 'going-to-maintenance' and a timeout [#timeout]_.
265 After receiving the MaintenanceRequest,the VIM decides on the actions to be taken
266 based on maintenance policies predefined by the affected Consumer(s).
268 .. [#timeout] Timeout is set by the Administrator and corresponds to the maximum time
269 to empty the physical resources.
271 .. figure:: images/figure5a.png
275 High-level message flow for maintenance policy enforcement
277 The high level message flow for the NFVI maintenance policy enforcement is shown
278 in :numref:`figure5a`. It consists of the following steps:
280 1. Maintenance trigger received from Administrator.
281 2. VIM switches the affected physical resources to "going-to-maintenance" state e.g. so that no new
282 VM will be scheduled on the physical servers.
283 3. Database lookup to find the Consumer(s) and virtual resources affected by the maintenance
285 4. Maintenance policies are enforced in the VIM, e.g. affected VM(s) are shut down
286 on the physical server(s), or affected Consumer(s) are notified about the planned
287 maintenance operation (steps 4a/4b).
290 Once the affected Consumer(s) have been notified, they take specific actions (e.g. switch to standby
291 (STBY) configuration, request to terminate the virtual resource(s)) to allow the maintenance
292 action to be executed. After the physical resources have been emptied, the VIM puts the physical
293 resources in "in-maintenance" state and sends a MaintenanceResponse back to the Administrator.
295 .. figure:: images/figure5b.png
299 Successful NFVI maintenance
301 The high level message flow for a successful NFVI maintenance is show in :numref:`figure5b`.
302 It consists of the following steps:
304 5. The Consumer C3 switches to standby configuration (STBY).
305 6. Instructions from Consumers C2/C3 are shared to VIM requesting certain actions to be performed
306 (steps 6a, 6b). After receiving such instructions, the VIM executes the requested
307 action in order to empty the physical resources (step 6c) and informs the
308 Consumer about the result of the actions (steps 6d, 6e).
309 7. The VIM switches the physical resources to "in-maintenance" state
310 8. Maintenance response is sent from VIM to inform the Administrator that the physical
311 servers have been emptied.
312 9. The Administrator is coordinating and executing the maintenance
313 operation/work on the NFVI. Note: this step is out of scope of Doctor project.
315 The requested actions to empty the physical resources may not be successful (e.g. migration fails
316 or takes too long) and in such a case, the VIM puts the physical resources back to 'enabled' and
317 informs the Administrator about the problem.
319 .. figure:: images/figure5c.png
323 Example of failed NFVI maintenance
325 An example of a high level message flow to cover the failed NFVI maintenance case is
326 shown in :numref:`figure5c`.
327 It consists of the following steps:
329 5. The Consumer C3 switches to standby configuration (STBY).
330 6. Instructions from Consumers C2/C3 are shared to VIM requesting certain actions to be performed
331 (steps 6a, 6b). The VIM executes the requested actions and sends back a NACK to consumer C2
332 (step 6d) as the migration of the virtual resource(s) is not completed by the given timeout.
333 7. The VIM switches the physical resources to "enabled" state.
334 8. MaintenanceNotification is sent from VIM to inform the Administrator that the maintenance action
339 vim: set tabstop=4 expandtab textwidth=80:
Code Review / doctor.git / blob