1 ====================================================
2 Report host fault to update server state immediately
3 ====================================================
5 https://blueprints.launchpad.net/nova/+spec/update-server-state-immediately
7 A new API is needed to report a host fault to change the state of the
8 instances and compute node immediately. This allows usage of evacuate API
9 without a delay. The new API provides the possibility for external monitoring
10 system to detect any kind of host failure fast and reliably and inform
11 OpenStack about it. Nova updates the compute node state and states of the
12 instances. This way the states in the Nova DB will be in sync with the
13 real state of the system.
17 * Nova state change for failed or unreachable host is slow and does not
18 reliably state compute node is down or not. This might cause same instance
19 to run twice if action taken to evacuate instance to another host.
20 * Nova state for instances on failed compute node will not change,
21 but remains active and running. This gives user a false information about
22 instance state. Currently one would need to call "nova reset-state" for each
23 instance to have them in error state.
24 * OpenStack user cannot make HA actions fast and reliably by trusting instance
25 state and compute node state.
26 * As compute node state changes slowly one cannot evacuate instances.
30 Use case in general is that in case there is a host fault one should change
31 compute node state fast and reliably when using DB servicegroup backend.
32 On top of this here is the use cases that are not covered currently to have
33 instance states changed correctly:
34 * Management network connectivity lost between controller and compute node.
37 Generic use case flow:
39 * The external monitoring system detects a host fault.
40 * The external monitoring system fences the host if not down already.
41 * The external system calls the new Nova API to force the failed compute node
42 into down state as well as instances running on it.
43 * Nova updates the compute node state and state of the effected instances to
46 Currently nova-compute state will be changing "down", but it takes a long
47 time. Server state keeps as "vm_state: active" and "power_state:
48 running", which is not correct. By having external tool to detect host faults
49 fast, fence host by powering down and then report host down to OpenStack, all
50 these states would reflect to actual situation. Also if OpenStack will not
51 implement automatic actions for fault correlation, external tool can do that.
52 This could be configured for example in server instance METADATA easily and be
53 read by external tool.
57 Liberty priorities have not yet been defined.
61 There needs to be a new API for Admin to state host is down. This API is used
62 to mark compute node and instances running on it down to reflect the real
65 Example on compute node is:
67 * When compute node is up and running:
68 vm_state: active and power_state: running
69 nova-compute state: up status: enabled
70 * When compute node goes down and new API is called to state host is down:
71 vm_state: stopped power_state: shutdown
72 nova-compute state: down status: enabled
74 vm_state values: soft-delete, deleted, resized and error
75 should not be touched.
76 task_state effect needs to be worked out if needs to be touched.
80 There is no attractive alternatives to detect all different host faults than
81 to have a external tool to detect different host faults. For this kind of tool
82 to exist there needs to be new API in Nova to report fault. Currently there
83 must have been some kind of workarounds implemented as cannot trust or get the
84 states from OpenStack fast enough.
92 * Update CLI to report host is down
94 nova host-update command
96 usage: nova host-update [--status <enable|disable>]
97 [--maintenance <enable|disable>]
101 Update host settings.
110 --status <enable|disable>
111 Either enable or disable a host.
113 --maintenance <enable|disable>
114 Either put or resume host to/from maintenance.
117 Report host down to update instance and compute node state in db.
119 * Update Compute API to report host is down:
121 /v2.1/{tenant_id}/os-hosts/{host_name}
123 Normal response codes: 200
126 Parameter Style Type Description
127 host_name URI xsd:string The name of the host of interest to you.
132 "maintenance_mode": "enable"
133 "host_down_reported": "true"
141 "host": "65c5d5b7e3bd44308e67fc50f362aee6",
142 "maintenance_mode": "enabled",
144 "host_down_reported": "true"
150 * New method to nova.compute.api module HostAPI class to have a
151 to mark host related instances and compute node down:
152 set_host_down(context, host_name)
154 * class novaclient.v2.hosts.HostManager(api) method update(host, values)
155 Needs to handle reporting host down.
157 * Schema does not need changes as in db only service and server states are to
162 API call needs admin privileges (in the default policy configuration).
168 Other end user impact
169 ---------------------
174 Only impact is that user can get information faster about instance and
175 compute node state. This also gives possibility to evacuate faster.
176 No impact that would slow down. Host down should be rare occurrence.
178 Other deployer impact
179 ---------------------
180 Developer can make use of any external tool to detect host fault and report it
191 Primary assignee: Tomi Juvonen
192 Other contributors: Ryota Mibu
206 Test cases that exists for enabling or putting host to maintenance should be
207 altered or similar new cases made test new functionality.
212 New API needs to be documented:
214 * Compute API extensions documentation.
215 http://developer.openstack.org/api-ref-compute-v2.1.html
216 * Nova commands documentation.
217 http://docs.openstack.org/user-guide-admin/content/novaclient_commands.html
218 * Compute command-line client documentation.
219 http://docs.openstack.org/cli-reference/content/novaclient_commands.html
220 * nova.compute.api documentation.
221 http://docs.openstack.org/developer/nova/api/nova.compute.api.html
222 * High Availability guide might have page to tell external tool could provide
223 ability to provide faster HA as able to update states by new API.
224 http://docs.openstack.org/high-availability-guide/content/index.html
228 * OPNFV Doctor project: https://wiki.opnfv.org/doctor
229 * OpenStack Instance HA Proposal:
230 http://blog.russellbryant.net/2014/10/15/openstack-instance-ha-proposal/
231 * The Different Facets of OpenStack HA:
232 http://blog.russellbryant.net/2015/03/10/
233 the-different-facets-of-openstack-ha/