1 Detailed architecture and interface specification
2 =================================================
4 This section describes a detailed implementation plan, which is based on the
5 high level architecture introduced in Section 3. Section 5.1 describes the
6 functional blocks of the Doctor architecture, which is followed by a high level
7 message flow in Section 5.2. Section 5.3 provides a mapping of selected existing
8 open source components to the building blocks of the Doctor architecture.
9 Thereby, the selection of components is based on their maturity and the gap
10 analysis executed in Section 4. Sections 5.4 and 5.5 detail the specification of
11 the related northbound interface and the related information elements. Finally,
12 Section 5.6 provides a first set of blueprints to address selected gaps required
13 for the realization functionalities of the Doctor project.
20 This section introduces the functional blocks to form the VIM. OpenStack was
21 selected as the candidate for implementation. Inside the VIM, 4 different
22 building blocks are defined (see :numref:`figure6`).
24 .. figure:: images/figure6.png
33 The Monitor module has the responsibility for monitoring the virtualized
34 infrastructure. There are already many existing tools and services (e.g. Zabbix)
35 to monitor different aspects of hardware and software resources which can be
36 used for this purpose.
41 The Inspector module has the ability a) to receive various failure notifications
42 regarding physical resource(s) from Monitor module(s), b) to find the affected
43 virtual resource(s) by querying the resource map in the Controller, and c) to
44 update the state of the virtual resource (and physical resource).
46 The Inspector has drivers for different types of events and resources to
47 integrate any type of Monitor and Controller modules. It also uses a failure
48 policy database to decide on the failure selection and aggregation from raw
49 events. This failure policy database is configured by the Administrator.
51 The reason for separation of the Inspector and Controller modules is to make the
52 Controller focus on simple operations by avoiding a tight integration of various
53 health check mechanisms into the Controller.
58 The Controller is responsible for maintaining the resource map (i.e. the mapping
59 from physical resources to virtual resources), accepting update requests for the
60 resource state(s) (exposing as provider API), and sending all failure events
61 regarding virtual resources to the Notifier. Optionally, the Controller has the
62 ability to force the state of a given physical resource to down in the resource
63 mapping when it receives failure notifications from the Inspector for that
64 given physical resource.
65 The Controller also re-calculates the capacity of the NVFI when receiving a
66 failure notification for a physical resource.
68 In a real-world deployment, the VIM may have several controllers, one for each
69 resource type, such as Nova, Neutron and Cinder in OpenStack. Each controller
70 maintains a database of virtual and physical resources which shall be the master
71 source for resource information inside the VIM.
76 The focus of the Notifier is on selecting and aggregating failure events
77 received from the controller based on policies mandated by the Consumer.
78 Therefore, it allows the Consumer to subscribe for alarms regarding virtual
79 resources using a method such as API endpoint. After receiving a fault
80 event from a Controller, it will notify the fault to the Consumer by referring
81 to the alarm configuration which was defined by the Consumer earlier on.
83 To reduce complexity of the Controller, it is a good approach for the
84 Controllers to emit all notifications without any filtering mechanism and have
85 another service (i.e. Notifier) handle those notifications properly. This is the
86 general philosophy of notifications in OpenStack. Note that a fault message
87 consumed by the Notifier is different from the fault message received by the
88 Inspector; the former message is related to virtual resources which are visible
89 to users with relevant ownership, whereas the latter is related to raw devices
90 or small entities which should be handled with an administrator privilege.
92 The northbound interface between the Notifier and the Consumer/Administrator is
93 specified in :ref:`impl_nbi`.
101 The detailed work flow for fault management is as follows (see also :numref:`figure7`):
103 1. Request to subscribe to monitor specific virtual resources. A query filter
104 can be used to narrow down the alarms the Consumer wants to be informed
106 2. Each subscription request is acknowledged with a subscribe response message.
107 The response message contains information about the subscribed virtual
108 resources, in particular if a subscribed virtual resource is in "alarm"
110 3. The NFVI sends monitoring events for resources the VIM has been subscribed
111 to. Note: this subscription message exchange between the VIM and NFVI is not
112 shown in this message flow.
113 4. Event correlation, fault detection and aggregation in VIM.
114 5. Database lookup to find the virtual resources affected by the detected fault.
115 6. Fault notification to Consumer.
116 7. The Consumer switches to standby configuration (STBY)
117 8. Instructions to VIM requesting certain actions to be performed on the
118 affected resources, for example migrate/update/terminate specific
119 resource(s). After reception of such instructions, the VIM is executing the
120 requested action, e.g. it will migrate or terminate a virtual resource.
121 a. Query request from Consumer to VIM to get information about the current
122 status of a resource.
123 b. Response to the query request with information about the current status of
124 the queried resource. In case the resource is in "fault" state, information
125 about the related fault(s) is returned.
127 In order to allow for quick reaction to failures, the time interval between
128 fault detection in step 3 and the corresponding recovery actions in step 7 and 8
129 shall be less than 1 second.
131 .. figure:: images/figure7.png
135 Fault management work flow
137 .. figure:: images/figure8.png
141 Fault management scenario
143 :numref:`figure8` shows a more detailed message flow (Steps 4 to 6) between
144 the 4 building blocks introduced in :ref:`impl_fb`.
146 4. The Monitor observed a fault in the NFVI and reports the raw fault to the
148 The Inspector filters and aggregates the faults using pre-configured
152 a) The Inspector queries the Resource Map to find the virtual resources
153 affected by the raw fault in the NFVI.
154 b) The Inspector updates the state of the affected virtual resources in the
156 c) The Controller observes a change of the virtual resource state and informs
157 the Notifier about the state change and the related alarm(s).
158 Alternatively, the Inspector may directly inform the Notifier about it.
160 6. The Notifier is performing another filtering and aggregation of the changes
161 and alarms based on the pre-configured alarm configuration. Finally, a fault
162 notification is sent to northbound to the Consumer.
166 .. figure:: images/figure9.png
170 NFVI maintenance work flow
172 The detailed work flow for NFVI maintenance is shown in :numref:`figure9`
173 and has the following steps. Note that steps 1, 2, and 5 to 8a in the NFVI
174 maintenance work flow are very similar to the steps in the fault management work
175 flow and share a similar implementation plan in Release 1.
177 1. Subscribe to fault/maintenance notifications.
178 2. Response to subscribe request.
179 3. Maintenance trigger received from administrator.
180 4. VIM switches NFVI resources to "maintenance" state. This, e.g., means they
181 should not be used for further allocation/migration requests
182 5. Database lookup to find the virtual resources affected by the detected
183 maintenance operation.
184 6. Maintenance notification to Consumer.
185 7. The Consumer switches to standby configuration (STBY)
186 8. Instructions from Consumer to VIM requesting certain recovery actions to be
187 performed (step 8a). After reception of such instructions, the VIM is
188 executing the requested action in order to empty the physical resources (step
190 9. Maintenance response from VIM to inform the Administrator that the physical
191 machines have been emptied (or the operation resulted in an error state).
192 10. Administrator is coordinating and executing the maintenance operation/work
194 a) Query request from Administrator to VIM to get information about the
195 current state of a resource.
196 b) Response to the query request with information about the current state of
197 the queried resource(s). In case the resource is in "maintenance" state,
198 information about the related maintenance operation is returned.
200 .. figure:: images/figure10.png
204 NFVI Maintenance implementation plan
206 :numref:`figure10` shows a more detailed message flow (Steps 3 to 6 and 9)
207 between the 4 building blocks introduced in Section 5.1..
209 3. The Administrator is sending a StateChange request to the Controller residing
211 4. The Controller queries the Resource Map to find the virtual resources
212 affected by the planned maintenance operation.
215 a) The Controller updates the state of the affected virtual resources in the
216 Resource Map database.
218 b) The Controller informs the Notifier about the virtual resources that will
219 be affected by the maintenance operation.
221 6. A maintenance notification is sent to northbound to the Consumer.
225 9. The Controller informs the Administrator after the physical resources have
230 Implementation plan for OPNFV Release 1
231 ---------------------------------------
236 :numref:`figure11` shows the implementation plan based on OpenStack and
237 related components as planned for Release 1. Hereby, the Monitor can be realized
238 by Zabbix. The Controller is realized by OpenStack Nova [NOVA]_, Neutron
239 [NEUT]_, and Cinder [CIND]_ for compute, network, and storage,
240 respectively. The Inspector can be realized by Monasca [MONA]_ or a simple
241 script querying Nova in order to map between physical and virtual resources. The
242 Notifier will be realized by Ceilometer [CEIL]_ receiving failure events
243 on its notification bus.
245 :numref:`figure12` shows the inner-workings of Ceilometer. After receiving
246 an "event" on its notification bus, first a notification agent will grab the
247 event and send a "notification" to the Collector. The collector writes the
248 notifications received to the Ceilometer databases.
250 In the existing Ceilometer implementation, an alarm evaluator is periodically
251 polling those databases through the APIs provided. If it finds new alarms, it
252 will evaluate them based on the pre-defined alarm configuration, and depending
253 on the configuration, it will hand a message to the Alarm Notifier, which in
254 turn will send the alarm message northbound to the Consumer. :numref:`figure12`
255 also shows an optimized work flow for Ceilometer with the goal to
256 reduce the delay for fault notifications to the Consumer. The approach is to
257 implement a new notification agent (called "publisher" in Ceilometer
258 terminology) which is directly sending the alarm through the "Notification Bus"
259 to a new "Notification-driven Alarm Evaluator (NAE)" (see Sections 5.6.2 and
260 5.6.3), thereby bypassing the Collector and avoiding the additional delay of the
261 existing polling-based alarm evaluator. The NAE is similar to the OpenStack
262 "Alarm Evaluator", but is triggered by incoming notifications instead of
263 periodically polling the OpenStack "Alarms" database for new alarms. The
264 Ceilometer "Alarms" database can hold three states: "normal", "insufficient
265 data", and "fired". It is representing a persistent alarm database. In order to
266 realize the Doctor requirements, we need to define new "meters" in the database
269 .. figure:: images/figure11.png
273 Implementation plan in OpenStack (OPNFV Release 1 ”Arno”)
276 .. figure:: images/figure12.png
280 Implementation plan in Ceilometer architecture
286 For NFVI Maintenance, a quite similar implementation plan exists. Instead of a
287 raw fault being observed by the Monitor, the Administrator is sending a
288 Maintenance Request through the northbound interface towards the Controller
289 residing in the VIM. Similar to the Fault Management use case, the Controller
290 (in our case OpenStack Nova) will send a maintenance event to the Notifier (i.e.
291 Ceilometer in our implementation). Within Ceilometer, the same workflow as
292 described in the previous section applies. In addition, the Controller(s) will
293 take appropriate actions to evacuate the physical machines in order to prepare
294 them for the planned maintenance operation. After the physical machines are
295 emptied, the Controller will inform the Administrator that it can initiate the
296 maintenance. Alternatively the VMs can just be shut down and boot up on the
297 same host after maintenance is over. There needs to be policy for administrator
298 to know the plan for VMs in maintenance.
303 This section introduces all attributes and information elements used in the
304 messages exchange on the northbound interfaces between the VIM and the VNFO and
307 Note: The information elements will be aligned with current work in ETSI NFV IFA
311 Simple information elements:
313 * SubscriptionID: identifies a subscription to receive fault or maintenance
315 * NotificationID: identifies a fault or maintenance notification.
316 * VirtualResourceID (Identifier): identifies a virtual resource affected by a
317 fault or a maintenance action of the underlying physical resource.
318 * PhysicalResourceID (Identifier): identifies a physical resource affected by a
319 fault or maintenance action.
320 * VirtualResourceState (String): state of a virtual resource, e.g. "normal",
321 "maintenance", "down", "error".
322 * PhysicalResourceState (String): state of a physical resource, e.g. "normal",
323 "maintenance", "down", "error".
324 * VirtualResourceType (String): type of the virtual resource, e.g. "virtual
325 machine", "virtual memory", "virtual storage", "virtual CPU", or "virtual
327 * FaultID (Identifier): identifies the related fault in the underlying physical
328 resource. This can be used to correlate different fault notifications caused
329 by the same fault in the physical resource.
330 * FaultType (String): Type of the fault. The allowed values for this parameter
331 depend on the type of the related physical resource. For example, a resource
332 of type "compute hardware" may have faults of type "CPU failure", "memory
333 failure", "network card failure", etc.
334 * Severity (Integer): value expressing the severity of the fault. The higher the
335 value, the more severe the fault.
336 * MinSeverity (Integer): value used in filter information elements. Only faults
337 with a severity higher than the MinSeverity value will be notified to the
339 * EventTime (Datetime): Time when the fault was observed.
340 * EventStartTime and EventEndTime (Datetime): Datetime range that can be used in
341 a FaultQueryFilter to narrow down the faults to be queried.
342 * ProbableCause: information about the probable cause of the fault.
343 * CorrelatedFaultID (Integer): list of other faults correlated to this fault.
344 * isRootCause (Boolean): Parameter indicating if this fault is the root for
345 other correlated faults. If TRUE, then the faults listed in the parameter
346 CorrelatedFaultID are caused by this fault.
347 * FaultDetails (Key-value pair): provides additional information about the
348 fault, e.g. information about the threshold, monitored attributes, indication
349 of the trend of the monitored parameter.
350 * FirmwareVersion (String): current version of the firmware of a physical
352 * HypervisorVersion (String): current version of a hypervisor.
353 * ZoneID (Identifier): Identifier of the resource zone. A resource zone is the
354 logical separation of physical and software resources in an NFVI deployment
355 for physical isolation, redundancy, or administrative designation.
356 * Metadata (Key-Value-Pairs): provides additional information of a physical
357 resource in maintenance/error state.
359 Complex information elements (see also UML diagrams in :numref:`figure13`
360 and :numref:`figure14`):
362 * VirtualResourceInfoClass:
364 + VirtualResourceID [1] (Identifier)
365 + VirtualResourceState [1] (String)
366 + Faults [0..*] (FaultClass): For each resource, all faults
367 including detailed information about the faults are provided.
369 * FaultClass: The parameters of the FaultClass are partially based on ETSI TS
370 132 111-2 (V12.1.0) [*]_, which is specifying fault management in 3GPP, in
371 particular describing the information elements used for alarm notifications.
373 - FaultID [1] (Identifier)
375 - Severity [1] (Integer)
376 - EventTime [1] (Datetime)
378 - CorrelatedFaultID [0..*] (Identifier)
379 - FaultDetails [0..*] (Key-value pair)
381 .. [*] http://www.etsi.org/deliver/etsi_ts/132100_132199/13211102/12.01.00_60/ts_13211102v120100p.pdf
383 * SubscribeFilterClass
385 - VirtualResourceType [0..*] (String)
386 - VirtualResourceID [0..*] (Identifier)
387 - FaultType [0..*] (String)
388 - MinSeverity [0..1] (Integer)
390 * FaultQueryFilterClass: narrows down the FaultQueryRequest, for example it
391 limits the query to certain physical resources, a certain zone, a given fault
392 type/severity/cause, or a specific FaultID.
394 - VirtualResourceType [0..*] (String)
395 - VirtualResourceID [0..*] (Identifier)
396 - FaultType [0..*] (String)
397 - MinSeverity [0..1] (Integer)
398 - EventStartTime [0..1] (Datetime)
399 - EventEndTime [0..1] (Datetime)
401 * PhysicalResourceStateClass:
403 - PhysicalResourceID [1] (Identifier)
404 - PhysicalResourceState [1] (String): mandates the new state of the physical
407 * PhysicalResourceInfoClass:
409 - PhysicalResourceID [1] (Identifier)
410 - PhysicalResourceState [1] (String)
411 - FirmwareVersion [0..1] (String)
412 - HypervisorVersion [0..1] (String)
413 - ZoneID [0..1] (Identifier)
415 * StateQueryFilterClass: narrows down a StateQueryRequest, for example it limits
416 the query to certain physical resources, a certain zone, or a given resource
417 state (e.g., only resources in "maintenance" state).
419 - PhysicalResourceID [1] (Identifier)
420 - PhysicalResourceState [1] (String)
421 - ZoneID [0..1] (Identifier)
425 Detailed northbound interface specification
426 -------------------------------------------
428 This section is specifying the northbound interfaces for fault management and
429 NFVI maintenance between the VIM on the one end and the Consumer and the
430 Administrator on the other ends. For each interface all messages and related
431 information elements are provided.
433 Note: The interface definition will be aligned with current work in ETSI NFV IFA
436 All of the interfaces described below are produced by the VIM and consumed by
437 the Consumer or Administrator.
439 Fault management interface
440 ^^^^^^^^^^^^^^^^^^^^^^^^^^
442 This interface allows the VIM to notify the Consumer about a virtual resource
443 that is affected by a fault, either within the virtual resource itself or by the
444 underlying virtualization infrastructure. The messages on this interface are
445 shown in :numref:`figure13` and explained in detail in the following
448 Note: The information elements used in this section are described in detail in
451 .. figure:: images/figure13.png
455 Fault management NB I/F messages
458 SubscribeRequest (Consumer -> VIM)
459 __________________________________
461 Subscription from Consumer to VIM to be notified about faults of specific
462 resources. The faults to be notified about can be narrowed down using a
467 - SubscribeFilter [1] (SubscribeFilterClass): Optional information to narrow
468 down the faults that shall be notified to the Consumer, for example limit to
469 specific VirtualResourceID(s), severity, or cause of the alarm.
471 SubscribeResponse (VIM -> Consumer)
472 ___________________________________
474 Response to a subscribe request message including information about the
475 subscribed resources, in particular if they are in "fault/error" state.
479 * SubscriptionID [1] (Identifier): Unique identifier for the subscription. It
480 can be used to delete or update the subscription.
481 * VirtualResourceInfo [0..*] (VirtualResourceInfoClass): Provides additional
482 information about the subscribed resources, i.e., a list of the related
483 resources, the current state of the resources, etc.
485 FaultNotification (VIM -> Consumer)
486 ___________________________________
488 Notification about a virtual resource that is affected by a fault, either within
489 the virtual resource itself or by the underlying virtualization infrastructure.
490 After reception of this request, the Consumer will decide on the optimal
491 action to resolve the fault. This includes actions like switching to a hot
492 standby virtual resource, migration of the fault virtual resource to another
493 physical machine, termination of the faulty virtual resource and instantiation
494 of a new virtual resource in order to provide a new hot standby resource. In
495 some use cases the Consumer can leave virtual resources on failed host to be
496 booted up again after fault is recovered. Existing resource management
497 interfaces and messages between the Consumer and the VIM can be used for those
498 actions, and there is no need to define additional actions on the Fault
499 Management Interface.
503 * NotificationID [1] (Identifier): Unique identifier for the notification.
504 * VirtualResourceInfo [1..*] (VirtualResourceInfoClass): List of faulty
505 resources with detailed information about the faults.
507 FaultQueryRequest (Consumer -> VIM)
508 ___________________________________
510 Request to find out about active alarms at the VIM. A FaultQueryFilter can be
511 used to narrow down the alarms returned in the response message.
515 * FaultQueryFilter [1] (FaultQueryFilterClass): narrows down the
516 FaultQueryRequest, for example it limits the query to certain physical
517 resources, a certain zone, a given fault type/severity/cause, or a specific
520 FaultQueryResponse (VIM -> Consumer)
521 ____________________________________
523 List of active alarms at the VIM matching the FaultQueryFilter specified in the
528 * VirtualResourceInfo [0..*] (VirtualResourceInfoClass): List of faulty
529 resources. For each resource all faults including detailed information about
530 the faults are provided.
535 The NFVI maintenance interfaces Consumer-VIM allows the Consumer to subscribe to
536 maintenance notifications provided by the VIM. The related maintenance interface
537 Administrator-VIM allows the Administrator to issue maintenance requests to the
538 VIM, i.e. requesting the VIM to take appropriate actions to empty physical
539 machine(s) in order to execute maintenance operations on them. The interface
540 also allows the Administrator to query the state of physical machines, e.g., in
541 order to get details in the current status of the maintenance operation like a
544 The messages defined in these northbound interfaces are shown in :numref:`figure14`
545 and described in detail in the following subsections.
547 .. figure:: images/figure14.png
551 NFVI maintenance NB I/F messages
553 SubscribeRequest (Consumer -> VIM)
554 __________________________________
556 Subscription from Consumer to VIM to be notified about maintenance operations
557 for specific virtual resources. The resources to be informed about can be
558 narrowed down using a subscribe filter.
562 * SubscribeFilter [1] (SubscribeFilterClass): Information to narrow down the
563 faults that shall be notified to the Consumer, for example limit to specific
564 virtual resource type(s).
566 SubscribeResponse (VIM -> Consumer)
567 ___________________________________
569 Response to a subscribe request message, including information about the
570 subscribed virtual resources, in particular if they are in "maintenance" state.
574 * SubscriptionID [1] (Identifier): Unique identifier for the subscription. It
575 can be used to delete or update the subscription.
576 * VirtualResourceInfo [0..*] (VirtalResourceInfoClass): Provides additional
577 information about the subscribed virtual resource(s), e.g., the ID, type and
578 current state of the resource(s).
580 MaintenanceNotification (VIM -> Consumer)
581 _________________________________________
583 Notification about a physical resource switched to "maintenance" state. After
584 reception of this request, the Consumer will decide on the optimal action to
585 address this request, e.g., to switch to the standby (STBY) configuration.
589 * VirtualResourceInfo [1..*] (VirtualResourceInfoClass): List of virtual
590 resources where the state has been changed to maintenance.
592 StateChangeRequest (Administrator -> VIM)
593 _________________________________________
595 Request to change the state of a list of physical resources, e.g. to
596 "maintenance" state, in order to prepare them for a planned maintenance
601 * PhysicalResourceState [1..*] (PhysicalResourceStateClass)
603 StateChangeResponse (VIM -> Administrator)
604 __________________________________________
606 Response message to inform the Administrator that the requested resources are
607 now in maintenance state (or the operation resulted in an error) and the
608 maintenance operation(s) can be executed.
612 * PhysicalResourceInfo [1..*] (PhysicalResourceInfoClass)
614 StateQueryRequest (Administrator -> VIM)
615 ________________________________________
617 In this procedure, the Administrator would like to get the information about
618 physical machine(s), e.g. their state ("normal", "maintenance"), firmware
619 version, hypervisor version, update status of firmware and hypervisor, etc. It
620 can be used to check the progress during firmware update and the confirmation
621 after update. A filter can be used to narrow down the resources returned in the
626 * StateQueryFilter [1] (StateQueryFilterClass): narrows down the
627 StateQueryRequest, for example it limits the query to certain physical
628 resources, a certain zone, or a given resource state.
630 StateQueryResponse (VIM -> Administrator)
631 _________________________________________
633 List of physical resources matching the filter specified in the
638 * PhysicalResourceInfo [0..*] (PhysicalResourceInfoClass): List of physical
639 resources. For each resource, information about the current state, the
640 firmware version, etc. is provided.
645 This section is listing a first set of blueprints that have been proposed by the
646 Doctor project to the open source community. Further blueprints addressing other
647 gaps identified in Section 4 will be submitted at a later stage of the OPNFV. In
648 this section the following definitions are used:
650 * "Event" is a message emitted by other OpenStack services such as Nova and
651 Neutron and is consumed by the "Notification Agents" in Ceilometer.
652 * "Notification" is a message generated by a "Notification Agent" in Ceilometer
653 based on an "event" and is delivered to the "Collectors" in Ceilometer that
654 store those notifications (as "sample") to the Ceilometer "Databases".
656 Instance State Notification (Ceilometer) [*]_
657 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
659 The Doctor project is planning to handle "events" and "notifications" regarding
660 Resource Status; Instance State, Port State, Host State, etc. Currently,
661 Ceilometer already receives "events" to identify the state of those resources,
662 but it does not handle and store them yet. This is why we also need a new event
663 definition to capture those resource states from "events" created by other
666 This BP proposes to add a new compute notification state to handle events from
667 an instance (server) from nova. It also creates a new meter "instance.state" in
670 .. [*] https://etherpad.opnfv.org/p/doctor_bps
672 Event Publisher for Alarm (Ceilometer) [*]_
673 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
675 **Problem statement:**
677 The existing "Alarm Evaluator" in OpenStack Ceilometer is periodically
678 querying/polling the databases in order to check all alarms independently from
679 other processes. This is adding additional delay to the fault notification
680 send to the Consumer, whereas one requirement of Doctor is to react on faults
683 The existing message flow is shown in :numref:`figure12`: after receiving
684 an "event", a "notification agent" (i.e. "event publisher") will send a
685 "notification" to a "Collector". The "collector" is collecting the
686 notifications and is updating the Ceilometer "Meter" database that is storing
687 information about the "sample" which is capured from original "event". The
688 "Alarm Evaluator" is periodically polling this databases then querying "Meter"
689 database based on each alarm configuration.
691 In the current Ceilometer implementation, there is no possibility to directly
692 trigger the "Alarm Evaluator" when a new "event" was received, but the "Alarm
693 Evaluator" will only find out that requires firing new notification to the
694 Consumer when polling the database.
696 **Change/feature request:**
698 This BP proposes to add a new "event publisher for alarm", which is bypassing
699 several steps in Ceilometer in order to avoid the polling-based approach of
700 the existing Alarm Evaluator that makes notification slow to users.
702 After receiving an "(alarm) event" by listening on the Ceilometer message
703 queue ("notification bus"), the new "event publisher for alarm" immediately
704 hands a "notification" about this event to a new Ceilometer component
705 "Notification-driven alarm evaluator" proposed in the other BP (see Section
708 Note, the term "publisher" refers to an entity in the Ceilometer architecture
709 (it is a "notification agent"). It offers the capability to provide
710 notifications to other services outside of Ceilometer, but it is also used to
711 deliver notifications to other Ceilometer components (e.g. the "Collectors")
712 via the Ceilometer "notification bus".
714 **Implementation detail**
716 * "Event publisher for alarm" is part of Ceilometer
717 * The standard AMQP message queue is used with a new topic string.
718 * No new interfaces have to be added to Ceilometer.
719 * "Event publisher for Alarm" can be configured by the Administrator of
720 Ceilometer to be used as "Notification Agent" in addition to the existing
722 * Existing alarm mechanisms of Ceilometer can be used allowing users to
723 configure how to distribute the "notifications" transformed from "events",
724 e.g. there is an option whether an ongoing alarm is re-issued or not
727 .. [*] https://etherpad.opnfv.org/p/doctor_bps
729 Notification-driven alarm evaluator (Ceilometer) [*]_
730 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
732 **Problem statement:**
734 The existing "Alarm Evaluator" in OpenStack Ceilometer is periodically
735 querying/polling the databases in order to check all alarms independently from
736 other processes. This is adding additional delay to the fault notification send
737 to the Consumer, whereas one requirement of Doctor is to react on faults as fast
740 **Change/feature request:**
742 This BP is proposing to add an alternative "Notification-driven Alarm Evaluator"
743 for Ceilometer that is receiving "notifications" sent by the "Event Publisher
744 for Alarm" described in the other BP. Once this new "Notification-driven Alarm
745 Evaluator" received "notification", it finds the "alarm" configurations which
746 may relate to the "notification" by querying the "alarm" database with some keys
747 i.e. resource ID, then it will evaluate each alarm with the information in that
750 After the alarm evaluation, it will perform the same way as the existing "alarm
751 evaluator" does for firing alarm notification to the Consumer. Similar to the
752 existing Alarm Evaluator, this new "Notification-driven Alarm Evaluator" is
753 aggregating and correlating different alarms which are then provided northbound
754 to the Consumer via the OpenStack "Alarm Notifier". The user/administrator can
755 register the alarm configuration via existing Ceilometer API [*]_. Thereby, he
756 can configure whether to set an alarm or not and where to send the alarms to.
758 **Implementation detail**
760 * The new "Notification-driven Alarm Evaluator" is part of Ceilometer.
761 * Most of the existing source code of the "Alarm Evaluator" can be re-used to
763 * No additional application logic is needed
764 * It will access the Ceilometer Databases just like the existing "Alarm
766 * Only the polling-based approach will be replaced by a listener for
767 "notifications" provided by the "Event Publisher for Alarm" on the Ceilometer
769 * No new interfaces have to be added to Ceilometer.
772 .. [*] https://etherpad.opnfv.org/p/doctor_bps
773 .. [*] https://wiki.openstack.org/wiki/Ceilometer/Alerting
775 Report host fault to update server state immediately (Nova) [*]_
776 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
778 **Problem statement:**
780 * Nova state change for failed or unreachable host is slow and does not reliably
781 state host is down or not. This might cause same server instance to run twice
782 if action taken to evacuate instance to another host.
783 * Nova state for server(s) on failed host will not change, but remains active
784 and running. This gives the user false information about server state.
785 * VIM northbound interface notification of host faults towards VNFM and NFVO
786 should be in line with OpenStack state. This fault notification is a Telco
787 requirement defined in ETSI and will be implemented by OPNFV Doctor project.
788 * Openstack user cannot make HA actions fast and reliably by trusting server
789 state and host state.
793 There needs to be a new API for Admin to state host is down. This API is used to
794 mark services running in host down to reflect the real situation.
796 Example on compute node is:
798 * When compute node is up and running:::
800 vm_state: activeand power_state: running
801 nova-compute state: up status: enabled
803 * When compute node goes down and new API is called to state host is down:::
805 vm_state: stopped power_state: shutdown
806 nova-compute state: down status: enabled
810 There is no attractive alternative to detect all different host faults than to
811 have an external tool to detect different host faults. For this kind of tool to
812 exist there needs to be new API in Nova to report fault. Currently there must be
813 some kind of workarounds implemented as cannot trust or get the states from
814 OpenStack fast enough.
816 .. [*] https://blueprints.launchpad.net/nova/+spec/update-server-state-immediately
821 This section lists some BPs related to Doctor, but proposed by drafters outside
824 pacemaker-servicegroup-driver [*]_
825 __________________________________
827 This BP will detect and report host down quite fast to OpenStack. This however
828 might not work properly for example when management network has some problem and
829 host reported faulty while VM still running there. This might lead to launching
830 same VM instance twice causing problems. Also NB IF message needs fault reason
831 and for that the source needs to be a tool that detects different kind of faults
832 as Doctor will be doing. Also this BP might need enhancement to change server
833 and service states correctly.
835 .. [*] https://blueprints.launchpad.net/nova/+spec/pacemaker-servicegroup-driver
838 vim: set tabstop=4 expandtab textwidth=80: