1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
7 OPNFV installers install most components of Doctor framework including
8 OpenStack Nova, Neutron and Cinder (Doctor Controller) and OpenStack
9 Ceilometer and Aodh (Doctor Notifier) except Doctor Monitor.
11 After major components of OPNFV are deployed, you can setup Doctor functions
12 by following instructions in this section. You can also learn detailed
13 steps in setup_installer() under `doctor/tests`_.
15 .. _doctor/tests: https://gerrit.opnfv.org/gerrit/gitweb?p=doctor.git;a=tree;f=tests;
20 You need to configure one of Doctor Inspector below.
22 **Doctor Sample Inspector**
24 Sample Inspector is intended to show minimum functions of Doctor Inspector.
26 Doctor Sample Inspector suggested to be placed in one of the controller nodes,
27 but it can be put on any host where Doctor Monitor can reach and access
28 the OpenStack Controller (Nova).
30 Make sure OpenStack env parameters are set properly, so that Doctor Inspector
31 can issue admin actions such as compute host force-down and state update of VM.
33 Then, you can configure Doctor Inspector as follows:
37 git clone https://gerrit.opnfv.org/gerrit/doctor -b stable/danube
40 python inspector.py $INSPECTOR_PORT > inspector.log 2>&1 &
44 OpenStack `Congress`_ is a Governance as a Service (previously Policy as a
45 Service). Congress implements Doctor Inspector as it can inspect a fault
46 situation and propagate errors onto other entities.
48 .. _Congress: https://wiki.openstack.org/wiki/Congress
50 Congress is deployed by OPNFV installers. You need to enable doctor
51 datasource driver and set policy rules. By the example configuration below,
52 Congress will force down nova compute service when it received a fault event
53 of that compute host. Also, Congress will set the state of all VMs running on
54 that host from ACTIVE to ERROR state.
58 openstack congress datasource create doctor doctor
60 openstack congress policy rule create \
61 --name host_down classification \
63 doctor:events(hostname=host, type="compute.host.down", status="down")'
65 openstack congress policy rule create \
66 --name active_instance_in_host classification \
67 'active_instance_in_host(vmid, host) :-
68 nova:servers(id=vmid, host_name=host, status="ACTIVE")'
70 openstack congress policy rule create \
71 --name host_force_down classification \
72 'execute[nova:services.force_down(host, "nova-compute", "True")] :-
75 openstack congress policy rule create \
76 --name error_vm_states classification \
77 'execute[nova:servers.reset_state(vmid, "error")] :-
79 active_instance_in_host(vmid, host)'
83 OpenStack `Vitrage`_ is an RCA (Root Cause Analysis) service for organizing,
84 analyzing and expanding OpenStack alarms & events. Vitrage implements Doctor
85 Inspector, as it receives a notification that a host is down and calls Nova
86 force-down API. In addition, it raises alarms on the instances running on this
89 .. _Vitrage: https://wiki.openstack.org/wiki/Vitrage
91 Vitrage is not deployed by OPNFV installers yet. It can be installed either on
92 top of a devstack environment, or on top of a real OpenStack environment. See
93 `Vitrage Installation`_
95 .. _`Vitrage Installation`: https://docs.openstack.org/developer/vitrage/installation-and-configuration.html
97 Doctor SB API and a Doctor datasource were implemented in Vitrage in the Ocata
98 release. The Doctor datasource is enabled by default.
100 After Vitrage is installed and configured, there is a need to configure it to
101 support the Doctor use case. This can be done in a few steps:
103 1. Make sure that 'aodh' and 'doctor' are included in the list of datasource
104 types in /etc/vitrage/vitrage.conf:
109 types = aodh,doctor,nova.host,nova.instance,nova.zone,static,cinder.volume,neutron.network,neutron.port,heat.stack
111 2. Enable the Vitrage Nova notifier. Set the following line in
112 /etc/vitrage/vitrage.conf:
119 3. Add a template that is responsible to call Nova force-down if Vitrage
120 receives a 'compute.host.down' alarm. Copy `template`_ and place it under
121 /etc/vitrage/templates
123 .. _template: https://github.com/openstack/vitrage/blob/master/etc/vitrage/templates.sample/host_down_scenarios.yaml
125 4. Restart the vitrage-graph and vitrage-notifier services
131 **Doctor Sample Monitor**
133 Doctor Monitors are suggested to be placed in one of the controller nodes,
134 but those can be put on any host which is reachable to target compute host and
135 accessible by the Doctor Inspector.
136 You need to configure Monitors for all compute hosts one by one.
138 Make sure OpenStack env parameters are set properly, so that Doctor Inspector
139 can issue admin actions such as compute host force-down and state update of VM.
141 Then, you can configure the Doctor Monitor as follows (Example for Apex deployment):
145 git clone https://gerrit.opnfv.org/gerrit/doctor -b stable/danube
148 COMPUTE_HOST='overcloud-novacompute-1.localdomain.com'
149 COMPUTE_IP=192.30.9.5
150 sudo python monitor.py "$COMPUTE_HOST" "$COMPUTE_IP" \
151 "http://127.0.0.1:$INSPECTOR_PORT/events" > monitor.log 2>&1 &