1 High level architecture and general features
2 ============================================
7 The Doctor project circles around two distinct use cases: 1) management of
8 failures of virtualized resources and 2) planned maintenance, e.g. migration, of
9 virtualized resources. Both of them may affect a VNF/application and the network
10 service it provides, but there is a difference in frequency and how they can be
13 Failures are spontaneous events that may or may not have an impact on the
14 virtual resources. The Consumer should as soon as possible react to the failure,
15 e.g., by switching to the STBY node. The Consumer will then instruct the VIM on
16 how to clean up or repair the lost virtual resources, i.e. restore the VM, VLAN
17 or virtualized storage. How much the applications are affected varies.
18 Applications with built-in HA support might experience a short decrease in
19 retainability (e.g. an ongoing session might be lost) while keeping availability
20 (establishment or re-establishment of sessions are not affected), whereas the
21 impact on applications without built-in HA may be more serious. How much the
22 network service is impacted depends on how the service is implemented. With
23 sufficient network redundancy the service may be unaffected even when a specific
26 On the other hand, planned maintenance impacting virtualized resources are events
27 that are known in advance. This group includes e.g. migration due to software
28 upgrades of OS and hypervisor on a compute host. Some of these might have been
29 requested by the application or its management solution, but there is also a
30 need for coordination on the actual operations on the virtual resources. There
31 may be an impact on the applications and the service, but since they are not
32 spontaneous events there is room for planning and coordination between the
33 application management organization and the infrastructure management
34 organization, including performing whatever actions that would be required to
35 minimize the problems.
37 Failure prediction is the process of pro-actively identifying situations that
38 may lead to a failure in the future unless acted on by means of maintenance
39 activities. From applications' point of view, failure prediction may impact them
40 in two ways: either the warning time is so short that the application or its
41 management solution does not have time to react, in which case it is equal to
42 the failure scenario, or there is sufficient time to avoid the consequences by
43 means of maintenance activities, in which case it is similar to planned
49 NFV and the Cloud platform provide virtual resources and related control
50 functionality to users and administrators. :num:`Figure #figure3` shows the high
51 level architecture of NFV focusing on the NFVI, i.e., the virtualized
52 infrastructure. The NFVI provides virtual resources, such as virtual machines
53 (VM) and virtual networks. Those virtual resources are used to run applications,
54 i.e. VNFs, which could be components of a network service which is managed by
55 the consumer of the NFVI. The VIM provides functionalities of controlling and
56 viewing virtual resources on hardware (physical) resources to the consumers,
57 i.e., users and administrators. OpenStack is a prominent candidate for this VIM.
58 The administrator may also directly control the NFVI without using the VIM.
60 Although OpenStack is the target upstream project where the new functional
61 elements (Controller, Notifier, Monitor, and Inspector) are expected to be
62 implemented, a particular implementation method is not assumed. Some of these
63 elements may sit outside of OpenStack and offer a northbound interface to
66 General Features and Requirements
67 ---------------------------------
69 The following features are required for the VIM to achieve high availability of
70 applications (e.g., MME, S/P-GW) and the Network Services:
72 1. Monitoring: Monitor physical and virtual resources.
73 2. Detection: Detect unavailability of physical resources.
74 3. Correlation and Cognition: Correlate faults and identify affected virtual
76 4. Notification: Notify unavailable virtual resources to their Consumer(s).
77 5. Fencing: Shut down or isolate a faulty resource
78 6. Recovery action: Execute actions to process fault recovery and maintenance.
80 The time interval between the instant that an event is detected by the
81 monitoring system and the Consumer notification of unavailable resources shall
82 be < 1 second (e.g., Step 1 to Step 4 in :num:`Figure #figure4` and :num:`Figure
87 .. figure:: images/figure3.png
90 High level architecture
95 The VIM shall monitor physical and virtual resources for unavailability and
101 The VIM shall detect unavailability and failures of physical resources that
102 might cause errors/faults in virtual resources running on top of them.
103 Unavailability of physical resource is detected by various monitoring and
104 managing tools for hardware and software components. This may include also
105 predicting upcoming faults. Note, fault prediction is out of scope of this
106 project and is investigated in the OPNFV "Data Collection for Failure
107 Prediction" project [PRED]_.
109 The fault items/events to be detected shall be configurable.
111 The configuration shall enable Failure Selection and Aggregation. Failure
112 aggregation means the VIM determines unavailability of physical resource from
113 more than two non-critical failures related to the same resource.
115 There are two types of unavailability - immediate and future:
117 * Immediate unavailability can be detected by setting traps of raw failures on
118 hardware monitoring tools.
119 * Future unavailability can be found by receiving maintenance instructions
120 issued by the administrator of the NFVI or by failure prediction mechanisms.
122 Correlation and Cognition
123 ^^^^^^^^^^^^^^^^^^^^^^^^^
125 The VIM shall correlate each fault to the impacted virtual resource, i.e., the
126 VIM shall identify unavailability of virtualized resources that are or will be
127 affected by failures on the physical resources under them. Unavailability of a
128 virtualized resource is determined by referring to the mapping of physical and
129 virtualized resources.
131 The relation from physical resources to virtualized resources shall be
132 configurable, as the cause of unavailability of virtualized resources can be
133 different in technologies and policies of deployment.
135 Failure aggregation is also required in this feature, e.g., a user may request
136 to be only notified if failures on more than two standby VMs in an (N+M)
137 deployment model occurred.
142 The VIM shall notify the alarm, i.e., unavailability of virtual resource(s), to
143 the Consumer owning it over the northbound interface, such that the Consumers
144 impacted by the failure can take appropriate actions to recover from the
147 The VIM shall also notify the unavailability of physical resources to its
150 All notifications shall be transferred immediately in order to minimize the
151 stalling time of the network service and to avoid over assignment caused by
152 delay of capability updates.
154 There may be multiple consumers, so the VIM has to find out the owner of a
155 faulty resource. Moreover, there may be a large number of virtual and physical
156 resources in a real deployment, so polling the state of all resources to the VIM
157 would lead to heavy signaling traffic. Thus, a publication/subscription
158 messaging model is better suited for these notifications, as notifications are
159 only sent to subscribed consumers.
161 Note: the VIM should only accept individual notification URLs for each resource
162 by its owner or administrator.
164 Notifications to the Consumer about the unavailability of virtualized
165 resources will include a description of the fault, preferably with sufficient
166 abstraction rather than detailed physical fault information. Flexibility in
167 notifications is important. For example, the receiver function in the
168 consumer-side implementation could have different schema, location, and policies
169 (e.g. receive or not, aggregate events with the same cause, etc.).
175 Recovery actions, e.g. safe VM evacuation, have to be preceded by fencing the
176 failed host. Fencing hereby means to isolate or shut down a faulty resource.
177 Without fencing -- when the perceived disconnection is due to some transient
178 or partial failure -- the evacuation might lead into two identical instances
179 running together and having a dangerous conflict.
181 There is a cross-project effort in OpenStack ongoing to implement fencing. A
182 general description of fencing in OpenStack is available here:
183 https://wiki.openstack.org/wiki/Fencing_Instances_of_an_Unreachable_Host .
188 In the basic "Fault management using ACT-STBY configuration" use case, no
189 automatic actions will be taken by the VIM, but all recovery actions executed by
190 the VIM and the NFVI will be instructed and coordinated by the Consumer.
192 In a more advanced use case, the VIM shall be able to recover the failed virtual
193 resources according to a pre-defined behavior for that resource. In principle
194 this means that the owner of the resource (i.e., its consumer or administrator)
195 can define which recovery actions shall be taken by the VIM. Examples are a
196 restart of the VM, migration/evacuation of the VM, or no action.
200 High level northbound interface specification
201 ---------------------------------------------
206 This interface allows the Consumer to subscribe to fault notification from the
207 VIM. Using a filter, the Consumer can narrow down which faults should be
208 notified. A fault notification may trigger the Consumer to switch from ACT to
209 STBY configuration and initiate fault recovery actions. A fault query
210 request/response message exchange allows the Consumer to find out about active
211 alarms at the VIM. A filter can be used to narrow down the alarms returned in
212 the response message.
216 .. figure:: images/figure4.png
219 High-level message flow for fault management
221 The high level message flow for the fault management use case is shown in
222 :num:`Figure #figure4`.
223 It consists of the following steps:
225 1. The VIM monitors the physical and virtual resources and the fault management
226 workflow is triggered by a monitored fault event.
227 2. Event correlation, fault detection and aggregation in VIM. Note: this may
228 also happen after Step 3.
229 3. Database lookup to find the virtual resources affected by the detected fault.
230 4. Fault notification to Consumer.
231 5. The Consumer switches to standby configuration (STBY)
232 6. Instructions to VIM requesting certain actions to be performed on the
233 affected resources, for example migrate/update/terminate specific
234 resource(s). After reception of such instructions, the VIM is executing the
235 requested action, e.g., it will migrate or terminate a virtual resource.
240 The NFVI maintenance interface allows the Administrator to notify the VIM about
241 a planned maintenance operation on the NFVI. A maintenance operation may for
242 example be an update of the server firmware or the hypervisor. The
243 MaintenanceRequest message contains instructions to change the state of the
244 resource from 'normal' to 'maintenance'. After receiving the MaintenanceRequest,
245 the VIM will notify the Consumer about the planned maintenance operation,
246 whereupon the Consumer will switch to standby (STBY) configuration to allow the
247 maintenance action to be executed. After the request was executed successfully
248 (i.e., the physical resources have been emptied) or the operation resulted in an
249 error state, the VIM sends a MaintenanceResponse message back to the
254 .. figure:: images/figure5.png
257 High-level message flow for NFVI maintenance
259 The high level message flow for the NFVI maintenance use case is shown in
260 :num:`Figure #figure5`.
261 It consists of the following steps:
263 1. Maintenance trigger received from administrator.
264 2. VIM switches the affected NFVI resources to "maintenance" state, i.e., the
265 NFVI resources are prepared for the maintenance operation. For example, the
266 virtual resources should not be used for further allocation/migration
267 requests and the VIM will coordinate with the Consumer on how to best empty
268 the physical resources.
269 3. Database lookup to find the virtual resources affected by the detected
270 maintenance operation.
271 4. StateChange notification to inform Consumer about planned maintenance
273 5. The Consumer switches to standby configuration (STBY)
274 6. Instructions from Consumer to VIM requesting certain actions to be performed
275 (step 6a). After receiving such instructions, the VIM executes the requested
276 action in order to empty the physical resources (step 6b) and informs the
277 Consumer is about the result of the actions. Note: this step is out of scope
279 7. Maintenance response from VIM to inform the Administrator that the physical
280 machines have been emptied (or the operation resulted in an error state).
281 8. The Administrator is coordinating and executing the maintenance
282 operation/work on the NFVI. Note: this step is out of scope of Doctor.
287 Faults in the listed elements need to be immediately notified to the Consumer in
288 order to perform an immediate action like live migration or switch to a hot
289 standby entity. In addition, the Administrator of the host should trigger a
290 maintenance action to, e.g., reboot the server or replace a defective hardware
293 Faults can be of different severity, i.e., critical, warning, or
294 info. Critical faults require immediate action as a severe degradation of the
295 system has happened or is expected. Warnings indicate that the system
296 performance is going down: related actions include closer (e.g. more frequent)
297 monitoring of that part of the system or preparation for a cold migration to a
298 backup VM. Info messages do not require any action. We also consider a type
299 "maintenance", which is no real fault, but may trigger maintenance actions
300 like a re-boot of the server or replacement of a faulty, but redundant HW.
302 Faults can be gathered by, e.g., enabling SNMP and installing some open source
303 tools to catch and poll SNMP. When using for example Zabbix one can also put an
304 agent running on the hosts to catch any other fault. In any case of failure, the
305 Administrator should be notified. Table 1 provides a list of high level faults
306 that are considered within the scope of the Doctor project requiring immediate
307 action by the Consumer.
310 +------------------+---------------------------------------------------------------------------------------------------------------------------+------------------+-------------------+------------------------------------------------------------------------------------------+----------------------------------------------------------------------+
311 | Service | Fault | Severity | How to detect? | Comment | Action to recover |
312 +------------------+---------------------------------------------------------------------------------------------------------------------------+------------------+-------------------+------------------------------------------------------------------------------------------+----------------------------------------------------------------------+
313 | Compute Hardware | Processor/CPU failure, CPU condition not ok | Critical | Zabbix | | Switch to hot standby |
314 + +---------------------------------------------------------------------------------------------------------------------------+------------------+-------------------+------------------------------------------------------------------------------------------+----------------------------------------------------------------------+
315 | | Memory failure/Memory condition not ok | Critical | Zabbix (IPMI) | | Switch to hot standby |
316 + +---------------------------------------------------------------------------------------------------------------------------+------------------+-------------------+------------------------------------------------------------------------------------------+----------------------------------------------------------------------+
317 | | Network card failure, e.g. network adapter connectivity lost | Critical | Zabbix/Ceilometer | | Switch to hot standby |
318 + +---------------------------------------------------------------------------------------------------------------------------+------------------+-------------------+------------------------------------------------------------------------------------------+----------------------------------------------------------------------+
319 | | Disk crash | Info | RAID monitoring | Network storage is very redundant (e.g. RAID system) and can guarantee high availability | Inform OAM |
320 + +---------------------------------------------------------------------------------------------------------------------------+------------------+-------------------+------------------------------------------------------------------------------------------+----------------------------------------------------------------------+
321 | | Storage controller | Critical | Zabbix (IPMI) | | Live migration if storage is still accessible; otherwise hot standby |
322 + +---------------------------------------------------------------------------------------------------------------------------+------------------+-------------------+------------------------------------------------------------------------------------------+----------------------------------------------------------------------+
323 | | PDU/power failure, power off, server reset | Critical | Zabbix/Ceilometer | | Switch to hot standby |
324 + +---------------------------------------------------------------------------------------------------------------------------+------------------+-------------------+------------------------------------------------------------------------------------------+----------------------------------------------------------------------+
325 | | Power degradation, power redundancy lost, power threshold exceeded | Warning | SNMP | | Live migration |
326 + +---------------------------------------------------------------------------------------------------------------------------+------------------+-------------------+------------------------------------------------------------------------------------------+----------------------------------------------------------------------+
327 | | Chassis problem (.e.g fan degraded/failed, chassis power degraded), CPU fan problem, temperature/thermal condition not ok | Warning | SNMP | | Live migration |
328 + +---------------------------------------------------------------------------------------------------------------------------+------------------+-------------------+------------------------------------------------------------------------------------------+----------------------------------------------------------------------+
329 | | Mainboard failure | Critical | Zabbix (IPMI) | | Switch to hot standby |
330 + +---------------------------------------------------------------------------------------------------------------------------+------------------+-------------------+------------------------------------------------------------------------------------------+----------------------------------------------------------------------+
331 | | OS crash (e.g. kernel panic) | Critical | Zabbix | | Switch to hot standby |
332 +------------------+---------------------------------------------------------------------------------------------------------------------------+------------------+-------------------+------------------------------------------------------------------------------------------+----------------------------------------------------------------------+
333 | Hypervisor | System has restarted | Critical | Zabbix | | Switch to hot standby |
334 + +---------------------------------------------------------------------------------------------------------------------------+------------------+-------------------+------------------------------------------------------------------------------------------+----------------------------------------------------------------------+
335 | | Hypervisor failure | Warning/Critical | Zabbix/Ceilometer | | Evacuation/switch to hot standby |
336 + +---------------------------------------------------------------------------------------------------------------------------+------------------+-------------------+------------------------------------------------------------------------------------------+----------------------------------------------------------------------+
337 | | Zabbix/Ceilometer is unreachable | Warning | ? | | Live migration |
338 +------------------+---------------------------------------------------------------------------------------------------------------------------+------------------+-------------------+------------------------------------------------------------------------------------------+----------------------------------------------------------------------+
339 | Network | SDN/OpenFlow switch, controller degraded/failed | Critical | ? | | Switch to hot standby or reconfigure virtual network topology |
340 + +---------------------------------------------------------------------------------------------------------------------------+------------------+-------------------+------------------------------------------------------------------------------------------+----------------------------------------------------------------------+
341 | | Hardware failure of physical switch/router | Warning | SNMP | Redundancy of physical infrastructure is reduced or no longer available | Live migration if possible, otherwise evacuation |
342 +------------------+---------------------------------------------------------------------------------------------------------------------------+------------------+-------------------+------------------------------------------------------------------------------------------+----------------------------------------------------------------------+
345 vim: set tabstop=4 expandtab textwidth=80:
Code Review / doctor.git / blob