1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
4 Detailed architecture and interface specification
5 =================================================
7 This section describes a detailed implementation plan, which is based on the
8 high level architecture introduced in Section 3. Section 5.1 describes the
9 functional blocks of the Doctor architecture, which is followed by a high level
10 message flow in Section 5.2. Section 5.3 provides a mapping of selected existing
11 open source components to the building blocks of the Doctor architecture.
12 Thereby, the selection of components is based on their maturity and the gap
13 analysis executed in Section 4. Sections 5.4 and 5.5 detail the specification of
14 the related northbound interface and the related information elements. Finally,
15 Section 5.6 provides a first set of blueprints to address selected gaps required
16 for the realization functionalities of the Doctor project.
23 This section introduces the functional blocks to form the VIM. OpenStack was
24 selected as the candidate for implementation. Inside the VIM, 4 different
25 building blocks are defined (see :numref:`figure6`).
27 .. figure:: images/figure6.png
36 The Monitor module has the responsibility for monitoring the virtualized
37 infrastructure. There are already many existing tools and services (e.g. Zabbix)
38 to monitor different aspects of hardware and software resources which can be
39 used for this purpose.
44 The Inspector module has the ability a) to receive various failure notifications
45 regarding physical resource(s) from Monitor module(s), b) to find the affected
46 virtual resource(s) by querying the resource map in the Controller, and c) to
47 update the state of the virtual resource (and physical resource).
49 The Inspector has drivers for different types of events and resources to
50 integrate any type of Monitor and Controller modules. It also uses a failure
51 policy database to decide on the failure selection and aggregation from raw
52 events. This failure policy database is configured by the Administrator.
54 The reason for separation of the Inspector and Controller modules is to make the
55 Controller focus on simple operations by avoiding a tight integration of various
56 health check mechanisms into the Controller.
61 The Controller is responsible for maintaining the resource map (i.e. the mapping
62 from physical resources to virtual resources), accepting update requests for the
63 resource state(s) (exposing as provider API), and sending all failure events
64 regarding virtual resources to the Notifier. Optionally, the Controller has the
65 ability to force the state of a given physical resource to down in the resource
66 mapping when it receives failure notifications from the Inspector for that
67 given physical resource.
68 The Controller also re-calculates the capacity of the NVFI when receiving a
69 failure notification for a physical resource.
71 In a real-world deployment, the VIM may have several controllers, one for each
72 resource type, such as Nova, Neutron and Cinder in OpenStack. Each controller
73 maintains a database of virtual and physical resources which shall be the master
74 source for resource information inside the VIM.
79 The focus of the Notifier is on selecting and aggregating failure events
80 received from the controller based on policies mandated by the Consumer.
81 Therefore, it allows the Consumer to subscribe for alarms regarding virtual
82 resources using a method such as API endpoint. After receiving a fault
83 event from a Controller, it will notify the fault to the Consumer by referring
84 to the alarm configuration which was defined by the Consumer earlier on.
86 To reduce complexity of the Controller, it is a good approach for the
87 Controllers to emit all notifications without any filtering mechanism and have
88 another service (i.e. Notifier) handle those notifications properly. This is the
89 general philosophy of notifications in OpenStack. Note that a fault message
90 consumed by the Notifier is different from the fault message received by the
91 Inspector; the former message is related to virtual resources which are visible
92 to users with relevant ownership, whereas the latter is related to raw devices
93 or small entities which should be handled with an administrator privilege.
95 The northbound interface between the Notifier and the Consumer/Administrator is
96 specified in :ref:`impl_nbi`.
104 The detailed work flow for fault management is as follows (see also :numref:`figure7`):
106 1. Request to subscribe to monitor specific virtual resources. A query filter
107 can be used to narrow down the alarms the Consumer wants to be informed
109 2. Each subscription request is acknowledged with a subscribe response message.
110 The response message contains information about the subscribed virtual
111 resources, in particular if a subscribed virtual resource is in "alarm"
113 3. The NFVI sends monitoring events for resources the VIM has been subscribed
114 to. Note: this subscription message exchange between the VIM and NFVI is not
115 shown in this message flow.
116 4. Event correlation, fault detection and aggregation in VIM.
117 5. Database lookup to find the virtual resources affected by the detected fault.
118 6. Fault notification to Consumer.
119 7. The Consumer switches to standby configuration (STBY)
120 8. Instructions to VIM requesting certain actions to be performed on the
121 affected resources, for example migrate/update/terminate specific
122 resource(s). After reception of such instructions, the VIM is executing the
123 requested action, e.g. it will migrate or terminate a virtual resource.
124 a. Query request from Consumer to VIM to get information about the current
125 status of a resource.
126 b. Response to the query request with information about the current status of
127 the queried resource. In case the resource is in "fault" state, information
128 about the related fault(s) is returned.
130 In order to allow for quick reaction to failures, the time interval between
131 fault detection in step 3 and the corresponding recovery actions in step 7 and 8
132 shall be less than 1 second.
134 .. figure:: images/figure7.png
138 Fault management work flow
140 .. figure:: images/figure8.png
144 Fault management scenario
146 :numref:`figure8` shows a more detailed message flow (Steps 4 to 6) between
147 the 4 building blocks introduced in :ref:`impl_fb`.
149 4. The Monitor observed a fault in the NFVI and reports the raw fault to the
151 The Inspector filters and aggregates the faults using pre-configured
155 a) The Inspector queries the Resource Map to find the virtual resources
156 affected by the raw fault in the NFVI.
157 b) The Inspector updates the state of the affected virtual resources in the
159 c) The Controller observes a change of the virtual resource state and informs
160 the Notifier about the state change and the related alarm(s).
161 Alternatively, the Inspector may directly inform the Notifier about it.
163 6. The Notifier is performing another filtering and aggregation of the changes
164 and alarms based on the pre-configured alarm configuration. Finally, a fault
165 notification is sent to northbound to the Consumer.
169 .. figure:: images/figure9.png
173 NFVI maintenance work flow
175 The detailed work flow for NFVI maintenance is shown in :numref:`figure9`
176 and has the following steps. Note that steps 1, 2, and 5 to 8a in the NFVI
177 maintenance work flow are very similar to the steps in the fault management work
178 flow and share a similar implementation plan in Release 1.
180 1. Subscribe to fault/maintenance notifications.
181 2. Response to subscribe request.
182 3. Maintenance trigger received from administrator.
183 4. VIM switches NFVI resources to "maintenance" state. This, e.g., means they
184 should not be used for further allocation/migration requests
185 5. Database lookup to find the virtual resources affected by the detected
186 maintenance operation.
187 6. Maintenance notification to Consumer.
188 7. The Consumer switches to standby configuration (STBY)
189 8. Instructions from Consumer to VIM requesting certain recovery actions to be
190 performed (step 8a). After reception of such instructions, the VIM is
191 executing the requested action in order to empty the physical resources (step
193 9. Maintenance response from VIM to inform the Administrator that the physical
194 machines have been emptied (or the operation resulted in an error state).
195 10. Administrator is coordinating and executing the maintenance operation/work
197 a) Query request from Administrator to VIM to get information about the
198 current state of a resource.
199 b) Response to the query request with information about the current state of
200 the queried resource(s). In case the resource is in "maintenance" state,
201 information about the related maintenance operation is returned.
203 .. figure:: images/figure10.png
207 NFVI Maintenance scenario
209 :numref:`figure10` shows a more detailed message flow (Steps 3 to 6 and 9)
210 between the 4 building blocks introduced in Section 5.1..
212 3. The Administrator is sending a StateChange request to the Controller residing
214 4. The Controller queries the Resource Map to find the virtual resources
215 affected by the planned maintenance operation.
218 a) The Controller updates the state of the affected virtual resources in the
219 Resource Map database.
221 b) The Controller informs the Notifier about the virtual resources that will
222 be affected by the maintenance operation.
224 6. A maintenance notification is sent to northbound to the Consumer.
228 9. The Controller informs the Administrator after the physical resources have
235 This section introduces all attributes and information elements used in the
236 messages exchange on the northbound interfaces between the VIM and the VNFO and
239 Note: The information elements will be aligned with current work in ETSI NFV IFA
243 Simple information elements:
245 * SubscriptionID (Identifier): identifies a subscription to receive fault or maintenance
247 * NotificationID (Identifier): identifies a fault or maintenance notification.
248 * VirtualResourceID (Identifier): identifies a virtual resource affected by a
249 fault or a maintenance action of the underlying physical resource.
250 * PhysicalResourceID (Identifier): identifies a physical resource affected by a
251 fault or maintenance action.
252 * VirtualResourceState (String): state of a virtual resource, e.g. "normal",
253 "maintenance", "down", "error".
254 * PhysicalResourceState (String): state of a physical resource, e.g. "normal",
255 "maintenance", "down", "error".
256 * VirtualResourceType (String): type of the virtual resource, e.g. "virtual
257 machine", "virtual memory", "virtual storage", "virtual CPU", or "virtual
259 * FaultID (Identifier): identifies the related fault in the underlying physical
260 resource. This can be used to correlate different fault notifications caused
261 by the same fault in the physical resource.
262 * FaultType (String): Type of the fault. The allowed values for this parameter
263 depend on the type of the related physical resource. For example, a resource
264 of type "compute hardware" may have faults of type "CPU failure", "memory
265 failure", "network card failure", etc.
266 * Severity (Integer): value expressing the severity of the fault. The higher the
267 value, the more severe the fault.
268 * MinSeverity (Integer): value used in filter information elements. Only faults
269 with a severity higher than the MinSeverity value will be notified to the
271 * EventTime (Datetime): Time when the fault was observed.
272 * EventStartTime and EventEndTime (Datetime): Datetime range that can be used in
273 a FaultQueryFilter to narrow down the faults to be queried.
274 * ProbableCause (String): information about the probable cause of the fault.
275 * CorrelatedFaultID (Integer): list of other faults correlated to this fault.
276 * isRootCause (Boolean): Parameter indicating if this fault is the root for
277 other correlated faults. If TRUE, then the faults listed in the parameter
278 CorrelatedFaultID are caused by this fault.
279 * FaultDetails (Key-value pair): provides additional information about the
280 fault, e.g. information about the threshold, monitored attributes, indication
281 of the trend of the monitored parameter.
282 * FirmwareVersion (String): current version of the firmware of a physical
284 * HypervisorVersion (String): current version of a hypervisor.
285 * ZoneID (Identifier): Identifier of the resource zone. A resource zone is the
286 logical separation of physical and software resources in an NFVI deployment
287 for physical isolation, redundancy, or administrative designation.
288 * Metadata (Key-value pair): provides additional information of a physical
289 resource in maintenance/error state.
291 Complex information elements (see also UML diagrams in :numref:`figure13`
292 and :numref:`figure14`):
294 * VirtualResourceInfoClass:
296 + VirtualResourceID [1] (Identifier)
297 + VirtualResourceState [1] (String)
298 + Faults [0..*] (FaultClass): For each resource, all faults
299 including detailed information about the faults are provided.
301 * FaultClass: The parameters of the FaultClass are partially based on ETSI TS
302 132 111-2 (V12.1.0) [*]_, which is specifying fault management in 3GPP, in
303 particular describing the information elements used for alarm notifications.
305 - FaultID [1] (Identifier)
306 - FaultType [1] (String)
307 - Severity [1] (Integer)
308 - EventTime [1] (Datetime)
309 - ProbableCause [1] (String)
310 - CorrelatedFaultID [0..*] (Identifier)
311 - FaultDetails [0..*] (Key-value pair)
313 .. [*] http://www.etsi.org/deliver/etsi_ts/132100_132199/13211102/12.01.00_60/ts_13211102v120100p.pdf
315 * SubscribeFilterClass
317 - VirtualResourceType [0..*] (String)
318 - VirtualResourceID [0..*] (Identifier)
319 - FaultType [0..*] (String)
320 - MinSeverity [0..1] (Integer)
322 * FaultQueryFilterClass: narrows down the FaultQueryRequest, for example it
323 limits the query to certain physical resources, a certain zone, a given fault
324 type/severity/cause, or a specific FaultID.
326 - VirtualResourceType [0..*] (String)
327 - VirtualResourceID [0..*] (Identifier)
328 - FaultType [0..*] (String)
329 - MinSeverity [0..1] (Integer)
330 - EventStartTime [0..1] (Datetime)
331 - EventEndTime [0..1] (Datetime)
333 * PhysicalResourceStateClass:
335 - PhysicalResourceID [1] (Identifier)
336 - PhysicalResourceState [1] (String): mandates the new state of the physical
338 - Metadata [0..*] (Key-value pair)
340 * PhysicalResourceInfoClass:
342 - PhysicalResourceID [1] (Identifier)
343 - PhysicalResourceState [1] (String)
344 - FirmwareVersion [0..1] (String)
345 - HypervisorVersion [0..1] (String)
346 - ZoneID [0..1] (Identifier)
347 - Metadata [0..*] (Key-value pair)
349 * StateQueryFilterClass: narrows down a StateQueryRequest, for example it limits
350 the query to certain physical resources, a certain zone, or a given resource
351 state (e.g., only resources in "maintenance" state).
353 - PhysicalResourceID [1] (Identifier)
354 - PhysicalResourceState [1] (String)
355 - ZoneID [0..1] (Identifier)
359 Detailed northbound interface specification
360 -------------------------------------------
362 This section is specifying the northbound interfaces for fault management and
363 NFVI maintenance between the VIM on the one end and the Consumer and the
364 Administrator on the other ends. For each interface all messages and related
365 information elements are provided.
367 Note: The interface definition will be aligned with current work in ETSI NFV IFA
370 All of the interfaces described below are produced by the VIM and consumed by
371 the Consumer or Administrator.
373 Fault management interface
374 ^^^^^^^^^^^^^^^^^^^^^^^^^^
376 This interface allows the VIM to notify the Consumer about a virtual resource
377 that is affected by a fault, either within the virtual resource itself or by the
378 underlying virtualization infrastructure. The messages on this interface are
379 shown in :numref:`figure13` and explained in detail in the following
382 Note: The information elements used in this section are described in detail in
385 .. figure:: images/figure13.png
389 Fault management NB I/F messages
392 SubscribeRequest (Consumer -> VIM)
393 __________________________________
395 Subscription from Consumer to VIM to be notified about faults of specific
396 resources. The faults to be notified about can be narrowed down using a
401 - SubscribeFilter [1] (SubscribeFilterClass): Optional information to narrow
402 down the faults that shall be notified to the Consumer, for example limit to
403 specific VirtualResourceID(s), severity, or cause of the alarm.
405 SubscribeResponse (VIM -> Consumer)
406 ___________________________________
408 Response to a subscribe request message including information about the
409 subscribed resources, in particular if they are in "fault/error" state.
413 * SubscriptionID [1] (Identifier): Unique identifier for the subscription. It
414 can be used to delete or update the subscription.
415 * VirtualResourceInfo [0..*] (VirtualResourceInfoClass): Provides additional
416 information about the subscribed resources, i.e., a list of the related
417 resources, the current state of the resources, etc.
419 FaultNotification (VIM -> Consumer)
420 ___________________________________
422 Notification about a virtual resource that is affected by a fault, either within
423 the virtual resource itself or by the underlying virtualization infrastructure.
424 After reception of this request, the Consumer will decide on the optimal
425 action to resolve the fault. This includes actions like switching to a hot
426 standby virtual resource, migration of the fault virtual resource to another
427 physical machine, termination of the faulty virtual resource and instantiation
428 of a new virtual resource in order to provide a new hot standby resource. In
429 some use cases the Consumer can leave virtual resources on failed host to be
430 booted up again after fault is recovered. Existing resource management
431 interfaces and messages between the Consumer and the VIM can be used for those
432 actions, and there is no need to define additional actions on the Fault
433 Management Interface.
437 * NotificationID [1] (Identifier): Unique identifier for the notification.
438 * VirtualResourceInfo [1..*] (VirtualResourceInfoClass): List of faulty
439 resources with detailed information about the faults.
441 FaultQueryRequest (Consumer -> VIM)
442 ___________________________________
444 Request to find out about active alarms at the VIM. A FaultQueryFilter can be
445 used to narrow down the alarms returned in the response message.
449 * FaultQueryFilter [1] (FaultQueryFilterClass): narrows down the
450 FaultQueryRequest, for example it limits the query to certain physical
451 resources, a certain zone, a given fault type/severity/cause, or a specific
454 FaultQueryResponse (VIM -> Consumer)
455 ____________________________________
457 List of active alarms at the VIM matching the FaultQueryFilter specified in the
462 * VirtualResourceInfo [0..*] (VirtualResourceInfoClass): List of faulty
463 resources. For each resource all faults including detailed information about
464 the faults are provided.
469 The NFVI maintenance interfaces Consumer-VIM allows the Consumer to subscribe to
470 maintenance notifications provided by the VIM. The related maintenance interface
471 Administrator-VIM allows the Administrator to issue maintenance requests to the
472 VIM, i.e. requesting the VIM to take appropriate actions to empty physical
473 machine(s) in order to execute maintenance operations on them. The interface
474 also allows the Administrator to query the state of physical machines, e.g., in
475 order to get details in the current status of the maintenance operation like a
478 The messages defined in these northbound interfaces are shown in :numref:`figure14`
479 and described in detail in the following subsections.
481 .. figure:: images/figure14.png
485 NFVI maintenance NB I/F messages
487 SubscribeRequest (Consumer -> VIM)
488 __________________________________
490 Subscription from Consumer to VIM to be notified about maintenance operations
491 for specific virtual resources. The resources to be informed about can be
492 narrowed down using a subscribe filter.
496 * SubscribeFilter [1] (SubscribeFilterClass): Information to narrow down the
497 faults that shall be notified to the Consumer, for example limit to specific
498 virtual resource type(s).
500 SubscribeResponse (VIM -> Consumer)
501 ___________________________________
503 Response to a subscribe request message, including information about the
504 subscribed virtual resources, in particular if they are in "maintenance" state.
508 * SubscriptionID [1] (Identifier): Unique identifier for the subscription. It
509 can be used to delete or update the subscription.
510 * VirtualResourceInfo [0..*] (VirtalResourceInfoClass): Provides additional
511 information about the subscribed virtual resource(s), e.g., the ID, type and
512 current state of the resource(s).
514 MaintenanceNotification (VIM -> Consumer)
515 _________________________________________
517 Notification about a physical resource switched to "maintenance" state. After
518 reception of this request, the Consumer will decide on the optimal action to
519 address this request, e.g., to switch to the standby (STBY) configuration.
523 * VirtualResourceInfo [1..*] (VirtualResourceInfoClass): List of virtual
524 resources where the state has been changed to maintenance.
526 StateChangeRequest (Administrator -> VIM)
527 _________________________________________
529 Request to change the state of a list of physical resources, e.g. to
530 "maintenance" state, in order to prepare them for a planned maintenance
535 * PhysicalResourceState [1..*] (PhysicalResourceStateClass)
537 StateChangeResponse (VIM -> Administrator)
538 __________________________________________
540 Response message to inform the Administrator that the requested resources are
541 now in maintenance state (or the operation resulted in an error) and the
542 maintenance operation(s) can be executed.
546 * PhysicalResourceInfo [1..*] (PhysicalResourceInfoClass)
548 StateQueryRequest (Administrator -> VIM)
549 ________________________________________
551 In this procedure, the Administrator would like to get the information about
552 physical machine(s), e.g. their state ("normal", "maintenance"), firmware
553 version, hypervisor version, update status of firmware and hypervisor, etc. It
554 can be used to check the progress during firmware update and the confirmation
555 after update. A filter can be used to narrow down the resources returned in the
560 * StateQueryFilter [1] (StateQueryFilterClass): narrows down the
561 StateQueryRequest, for example it limits the query to certain physical
562 resources, a certain zone, or a given resource state.
564 StateQueryResponse (VIM -> Administrator)
565 _________________________________________
567 List of physical resources matching the filter specified in the
572 * PhysicalResourceInfo [0..*] (PhysicalResourceInfoClass): List of physical
573 resources. For each resource, information about the current state, the
574 firmware version, etc. is provided.
576 NFV IFA, OPNFV Doctor and AODH alarms
577 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
579 This section compares the alarm interfaces of ETSI NFV IFA with the specifications
580 of this document and the alarm class of AODH.
582 ETSI NFV specifies an interface for alarms from virtualised resources in ETSI GS
583 NFV-IFA 005 [ENFV]_. The interface specifies an Alarm class and two notifications plus
584 operations to query alarm instances and to subscribe to the alarm notifications.
586 The specification in this document has a structure that is very similar to the
587 ETSI NFV specifications. The notifications differ in that an alarm notification
588 in the NFV interface defines a single fault for a single resource while the
589 notification specified in this document can contain multiple faults for
590 multiple resources. The Doctor specification is lacking the detailed time stamps
591 of the NFV specification essential for synchronizaion of the alarm list
592 using the query operation. The detailed time stamps are also of value in the event
593 and alarm history DBs.
595 AODH defines a base class for alarms, not the notifications. This means that
596 some of the dynamic attributes of the ETSI NFV alarm type, like alarmRaisedTime,
597 are not applicable to the AODH alarm class but are attributes of in the actual
598 notifications. (Description of these attributes will be added later.) The AODH alarm
599 class is lacking some attributes present in the NFV specification, fault details
600 and correlated alarms. Instead the AODH alarm class has attributes for actions,
601 rules and user and project id.
604 +------------------------+------------------------+---------------------+---------------------------------------------+---------------------------------------+
605 | ETSI NFV Alarm Type | OPNFV Doctor | AODH Event Alarm | Description / Comment | Recommendations |
606 | | Requirement Specs | Notification | | |
607 +========================+========================+=====================+=============================================+=======================================+
608 | alarmId | FaultId | alarm_id | Identifier of an alarm. | \- |
609 +------------------------+------------------------+---------------------+---------------------------------------------+---------------------------------------+
610 | \- | \- | alarm_name | Human readable alarm name. | May be added in ETSI NFV Stage 3. |
611 +------------------------+------------------------+---------------------+---------------------------------------------+---------------------------------------+
612 | managedObjectId | VirtualResourceId | (reason) | Identifier of the affected virtual resource | \- |
613 | | | | is part of the AODH reason parameter. | |
614 +------------------------+------------------------+---------------------+---------------------------------------------+---------------------------------------+
615 | \- | \- | user_id, project_id | User and project identifiers. | May be added in ETSI NFV Stage 3. |
616 +------------------------+------------------------+---------------------+---------------------------------------------+---------------------------------------+
617 | alarmRaisedTime | \- | \- | Timestamp when alarm was raised. | To be added to Doctor and AODH. May |
618 | | | | | be derived (e.g. in a shimlayer) from |
619 | | | | | the AODH alarm history. |
620 +------------------------+------------------------+---------------------+---------------------------------------------+---------------------------------------+
621 | alarmChangedTime | \- | \- | Timestamp when alarm was changed/updated. | see above |
622 +------------------------+------------------------+---------------------+---------------------------------------------+---------------------------------------+
623 | alarmClearedTime | \- | \- | Timestamp when alarm was cleared. | see above |
624 +------------------------+------------------------+---------------------+---------------------------------------------+---------------------------------------+
625 | eventTime | \- | \- | Timestamp when alarm was first observed by | see above |
626 | | | | the Monitor. | |
627 +------------------------+------------------------+---------------------+---------------------------------------------+---------------------------------------+
628 | \- | EventTime | generated | Timestamp of the Notification. | Update parameter name in Doctor spec. |
629 | | | | | May be added in ETSI NFV Stage 3. |
630 +------------------------+------------------------+---------------------+---------------------------------------------+---------------------------------------+
631 | state: | VirtualResourceState: | current: ok, alarm, | ETSI NFV IFA 005/006 lists example alarm | Maintenance state is missing in AODH. |
632 | E.g. Fired, Updated | E.g. normal, down | insufficient_data | states. | List of alarm states will be |
633 | Cleared | maintenance, error | | | specified in ETSI NFV Stage 3. |
634 +------------------------+------------------------+---------------------+---------------------------------------------+---------------------------------------+
635 | perceivedSeverity: | Severity (Integer) | Severity: | ETSI NFV IFA 005/006 lists example | List of alarm states will be |
636 | E.g. Critical, Major, | | low (default), | perceived severity values. | specified in ETSI NFV Stage 3. |
637 | Minor, Warning, | | moderate, critical | | |
638 | Indeterminate, Cleared | | | | **OPNFV: Severity (Integer)**: |
639 | | | | | * update OPNFV Doctor specification |
640 | | | | | to *Enum* |
642 | | | | | **perceivedSeverity=Indetermined**: |
643 | | | | | * remove value *Indetermined* in |
644 | | | | | IFA and map undefined values to |
645 | | | | | “minor” severity, or |
646 | | | | | * add value *indetermined* in AODH |
647 | | | | | and make it the default value. |
649 | | | | | **perceivedSeverity=Cleared**: |
650 | | | | | * remove value *Cleared* in IFA as |
651 | | | | | the information about a cleared |
652 | | | | | alarm alarm can be derived from |
653 | | | | | the alarm state parameter, or |
654 | | | | | * add value *cleared* in AODH and |
655 | | | | | set a rule that the severity is |
656 | | | | | “cleared” when the state is *ok*. |
657 +------------------------+------------------------+---------------------+---------------------------------------------+---------------------------------------+
658 | faultType | FaultType | event_type in | Type of the fault, e.g. “CPU failure” of a | OpenStack Alarming (Aodh) can use a |
659 | | | reason_data | compute resource, in machine interpretable | fuzzy matching with wildcard string, |
660 | | | | format. | "compute.cpu.failure". |
661 +------------------------+------------------------+---------------------+---------------------------------------------+---------------------------------------+
662 | N/A | N/A | type = "event" | Type of the notification. For fault | \- |
663 | | | | notifications the type in AODH is “event”. | |
664 +------------------------+------------------------+---------------------+---------------------------------------------+---------------------------------------+
665 | probableCause | ProbableCause | \- | Probable cause of the alarm. | May be provided (e.g. in a shimlayer) |
666 | | | | | based on Vitrage topology awareness / |
667 | | | | | root-cause-analysis. |
668 +------------------------+------------------------+---------------------+---------------------------------------------+---------------------------------------+
669 | isRootCause | IsRootCause | \- | Boolean indicating whether the fault is the | see above |
670 | | | | root cause of other faults. | |
671 +------------------------+------------------------+---------------------+---------------------------------------------+---------------------------------------+
672 | correlatedAlarmId | CorrelatedFaultId | \- | List of IDs of correlated faults. | see above |
673 +------------------------+------------------------+---------------------+---------------------------------------------+---------------------------------------+
674 | faultDetails | FaultDetails | \- | Additional details about the fault/alarm. | FaultDetails information element will |
675 | | | | | be specified in ETSI NFV Stage 3. |
676 +------------------------+------------------------+---------------------+---------------------------------------------+---------------------------------------+
677 | \- | \- | action, previous | Additional AODH alarm related parameters. | \- |
678 +------------------------+------------------------+---------------------+---------------------------------------------+---------------------------------------+
680 Table: Comparison of alarm attributes
682 The primary area of improvement should be alignment of the perceived severity. This
683 is important for a quick and accurate evaluation of the alarm. AODH thus should
684 support also the X.733 values Critical, Major, Minor, Warning and Indeterminate.
686 The detailed time stamps (raised, changed, cleared) which are essential for
687 synchronizing the alarm list using a query operation should be added to the
688 Doctor specification.
690 Other areas that need alignment is the so called alarm state in NFV. Here we must
691 however consider what can be attributes of the notification vs. what should be a
692 property of the alarm instance. This will be analyzed later.
696 Detailed southbound interface specification
697 -------------------------------------------
699 This section is specifying the southbound interfaces for fault management
700 between the Monitors and the Inspector.
701 Although southbound interfaces should be flexible to handle various events from
702 different types of Monitors, we define unified event API in order to improve
703 interoperability between the Monitors and the Inspector.
704 This is not limiting implementation of Monitor and Inspector as these could be
705 extended in order to support failures from intelligent inspection like prediction.
707 Note: The interface definition will be aligned with current work in ETSI NFV IFA
710 Fault event interface
711 ^^^^^^^^^^^^^^^^^^^^^
713 This interface allows the Monitors to notify the Inspector about an event which
714 was captured by the Monitor and may effect resources managed in the VIM.
720 Event notification including fault description.
721 The entity of this notification is event, and not fault or error specifically.
722 This allows us to use generic event format or framework build out of Doctor project.
723 The parameters below shall be mandatory, but keys in 'Details' can be optional.
727 * Time [1]: Datetime when the fault was observed in the Monitor.
728 * Type [1]: Type of event that will be used to process correlation in Inspector.
729 * Details [0..1]: Details containing additional information with Key-value pair style.
730 Keys shall be defined depending on the Type of the event.
738 'time': '2016-04-12T08:00:00',
739 'type': 'compute.host.down',
741 'hostname': 'compute-1',
742 'source': 'sample_monitor',
743 'cause': 'link-down',
744 'severity': 'critical',
746 'monitor_id': 'monitor-1',
747 'monitor_event_id': '123',
752 Optional parameters in 'Details':
754 * Hostname: the hostname on which the event occurred.
755 * Source: the display name of reporter of this event. This is not limited to monitor, other entity can be specified such as 'KVM'.
756 * Cause: description of the cause of this event which could be different from the type of this event.
757 * Severity: the severity of this event set by the monitor.
758 * Status: the status of target object in which error occurred.
759 * MonitorID: the ID of the monitor sending this event.
760 * MonitorEventID: the ID of the event in the monitor. This can be used by operator while tracking the monitor log.
761 * RelatedTo: the array of IDs which related to this event.
763 Also, we can have bulk API to receive multiple events in a single HTTP POST
764 message by using the 'events' wrapper as follows:
771 'time': '2016-04-12T08:00:00',
772 'type': 'compute.host.down',
776 'time': '2016-04-12T08:00:00',
777 'type': 'compute.host.nic.error',
789 This section is listing a first set of blueprints that have been proposed by the
790 Doctor project to the open source community. Further blueprints addressing other
791 gaps identified in Section 4 will be submitted at a later stage of the OPNFV. In
792 this section the following definitions are used:
794 * "Event" is a message emitted by other OpenStack services such as Nova and
795 Neutron and is consumed by the "Notification Agents" in Ceilometer.
796 * "Notification" is a message generated by a "Notification Agent" in Ceilometer
797 based on an "event" and is delivered to the "Collectors" in Ceilometer that
798 store those notifications (as "sample") to the Ceilometer "Databases".
800 Instance State Notification (Ceilometer) [*]_
801 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
803 The Doctor project is planning to handle "events" and "notifications" regarding
804 Resource Status; Instance State, Port State, Host State, etc. Currently,
805 Ceilometer already receives "events" to identify the state of those resources,
806 but it does not handle and store them yet. This is why we also need a new event
807 definition to capture those resource states from "events" created by other
810 This BP proposes to add a new compute notification state to handle events from
811 an instance (server) from nova. It also creates a new meter "instance.state" in
814 .. [*] https://etherpad.opnfv.org/p/doctor_bps
816 Event Publisher for Alarm (Ceilometer) [*]_
817 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
819 **Problem statement:**
821 The existing "Alarm Evaluator" in OpenStack Ceilometer is periodically
822 querying/polling the databases in order to check all alarms independently from
823 other processes. This is adding additional delay to the fault notification
824 send to the Consumer, whereas one requirement of Doctor is to react on faults
827 The existing message flow is shown in :numref:`figure12`: after receiving
828 an "event", a "notification agent" (i.e. "event publisher") will send a
829 "notification" to a "Collector". The "collector" is collecting the
830 notifications and is updating the Ceilometer "Meter" database that is storing
831 information about the "sample" which is capured from original "event". The
832 "Alarm Evaluator" is periodically polling this databases then querying "Meter"
833 database based on each alarm configuration.
835 .. figure:: images/figure12.png
839 Implementation plan in Ceilometer architecture
841 In the current Ceilometer implementation, there is no possibility to directly
842 trigger the "Alarm Evaluator" when a new "event" was received, but the "Alarm
843 Evaluator" will only find out that requires firing new notification to the
844 Consumer when polling the database.
846 **Change/feature request:**
848 This BP proposes to add a new "event publisher for alarm", which is bypassing
849 several steps in Ceilometer in order to avoid the polling-based approach of
850 the existing Alarm Evaluator that makes notification slow to users. See
853 After receiving an "(alarm) event" by listening on the Ceilometer message
854 queue ("notification bus"), the new "event publisher for alarm" immediately
855 hands a "notification" about this event to a new Ceilometer component
856 "Notification-driven alarm evaluator" proposed in the other BP (see Section
859 Note, the term "publisher" refers to an entity in the Ceilometer architecture
860 (it is a "notification agent"). It offers the capability to provide
861 notifications to other services outside of Ceilometer, but it is also used to
862 deliver notifications to other Ceilometer components (e.g. the "Collectors")
863 via the Ceilometer "notification bus".
865 **Implementation detail**
867 * "Event publisher for alarm" is part of Ceilometer
868 * The standard AMQP message queue is used with a new topic string.
869 * No new interfaces have to be added to Ceilometer.
870 * "Event publisher for Alarm" can be configured by the Administrator of
871 Ceilometer to be used as "Notification Agent" in addition to the existing
873 * Existing alarm mechanisms of Ceilometer can be used allowing users to
874 configure how to distribute the "notifications" transformed from "events",
875 e.g. there is an option whether an ongoing alarm is re-issued or not
878 .. [*] https://etherpad.opnfv.org/p/doctor_bps
880 Notification-driven alarm evaluator (Ceilometer) [*]_
881 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
883 **Problem statement:**
885 The existing "Alarm Evaluator" in OpenStack Ceilometer is periodically
886 querying/polling the databases in order to check all alarms independently from
887 other processes. This is adding additional delay to the fault notification send
888 to the Consumer, whereas one requirement of Doctor is to react on faults as fast
891 **Change/feature request:**
893 This BP is proposing to add an alternative "Notification-driven Alarm Evaluator"
894 for Ceilometer that is receiving "notifications" sent by the "Event Publisher
895 for Alarm" described in the other BP. Once this new "Notification-driven Alarm
896 Evaluator" received "notification", it finds the "alarm" configurations which
897 may relate to the "notification" by querying the "alarm" database with some keys
898 i.e. resource ID, then it will evaluate each alarm with the information in that
901 After the alarm evaluation, it will perform the same way as the existing "alarm
902 evaluator" does for firing alarm notification to the Consumer. Similar to the
903 existing Alarm Evaluator, this new "Notification-driven Alarm Evaluator" is
904 aggregating and correlating different alarms which are then provided northbound
905 to the Consumer via the OpenStack "Alarm Notifier". The user/administrator can
906 register the alarm configuration via existing Ceilometer API [*]_. Thereby, he
907 can configure whether to set an alarm or not and where to send the alarms to.
909 **Implementation detail**
911 * The new "Notification-driven Alarm Evaluator" is part of Ceilometer.
912 * Most of the existing source code of the "Alarm Evaluator" can be re-used to
914 * No additional application logic is needed
915 * It will access the Ceilometer Databases just like the existing "Alarm
917 * Only the polling-based approach will be replaced by a listener for
918 "notifications" provided by the "Event Publisher for Alarm" on the Ceilometer
920 * No new interfaces have to be added to Ceilometer.
923 .. [*] https://etherpad.opnfv.org/p/doctor_bps
924 .. [*] https://wiki.openstack.org/wiki/Ceilometer/Alerting
926 Report host fault to update server state immediately (Nova) [*]_
927 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
929 **Problem statement:**
931 * Nova state change for failed or unreachable host is slow and does not reliably
932 state host is down or not. This might cause same server instance to run twice
933 if action taken to evacuate instance to another host.
934 * Nova state for server(s) on failed host will not change, but remains active
935 and running. This gives the user false information about server state.
936 * VIM northbound interface notification of host faults towards VNFM and NFVO
937 should be in line with OpenStack state. This fault notification is a Telco
938 requirement defined in ETSI and will be implemented by OPNFV Doctor project.
939 * Openstack user cannot make HA actions fast and reliably by trusting server
940 state and host state.
944 There needs to be a new API for Admin to state host is down. This API is used to
945 mark services running in host down to reflect the real situation.
947 Example on compute node is:
949 * When compute node is up and running:::
951 vm_state: activeand power_state: running
952 nova-compute state: up status: enabled
954 * When compute node goes down and new API is called to state host is down:::
956 vm_state: stopped power_state: shutdown
957 nova-compute state: down status: enabled
961 There is no attractive alternative to detect all different host faults than to
962 have an external tool to detect different host faults. For this kind of tool to
963 exist there needs to be new API in Nova to report fault. Currently there must be
964 some kind of workarounds implemented as cannot trust or get the states from
965 OpenStack fast enough.
967 .. [*] https://blueprints.launchpad.net/nova/+spec/update-server-state-immediately
972 This section lists some BPs related to Doctor, but proposed by drafters outside
975 pacemaker-servicegroup-driver [*]_
976 __________________________________
978 This BP will detect and report host down quite fast to OpenStack. This however
979 might not work properly for example when management network has some problem and
980 host reported faulty while VM still running there. This might lead to launching
981 same VM instance twice causing problems. Also NB IF message needs fault reason
982 and for that the source needs to be a tool that detects different kind of faults
983 as Doctor will be doing. Also this BP might need enhancement to change server
984 and service states correctly.
986 .. [*] https://blueprints.launchpad.net/nova/+spec/pacemaker-servicegroup-driver