1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
5 This is a specification draft of a blueprint proposed for OpenStack Nova
6 Liberty. It was written by project member(s) and agreed within the project
7 before submitting it upstream. No further changes to its content will be
8 made here anymore; please follow it upstream:
10 * Current version upstream: https://review.openstack.org/#/c/169836/
11 * Development activity:
12 https://blueprints.launchpad.net/nova/+spec/mark-host-down
14 **Original draft is as follow:**
16 ====================================================
17 Report host fault to update server state immediately
18 ====================================================
20 https://blueprints.launchpad.net/nova/+spec/update-server-state-immediately
22 A new API is needed to report a host fault to change the state of the
23 instances and compute node immediately. This allows usage of evacuate API
24 without a delay. The new API provides the possibility for external monitoring
25 system to detect any kind of host failure fast and reliably and inform
26 OpenStack about it. Nova updates the compute node state and states of the
27 instances. This way the states in the Nova DB will be in sync with the
28 real state of the system.
32 * Nova state change for failed or unreachable host is slow and does not
33 reliably state compute node is down or not. This might cause same instance
34 to run twice if action taken to evacuate instance to another host.
35 * Nova state for instances on failed compute node will not change,
36 but remains active and running. This gives user a false information about
37 instance state. Currently one would need to call "nova reset-state" for each
38 instance to have them in error state.
39 * OpenStack user cannot make HA actions fast and reliably by trusting instance
40 state and compute node state.
41 * As compute node state changes slowly one cannot evacuate instances.
45 Use case in general is that in case there is a host fault one should change
46 compute node state fast and reliably when using DB servicegroup backend.
47 On top of this here is the use cases that are not covered currently to have
48 instance states changed correctly:
49 * Management network connectivity lost between controller and compute node.
52 Generic use case flow:
54 * The external monitoring system detects a host fault.
55 * The external monitoring system fences the host if not down already.
56 * The external system calls the new Nova API to force the failed compute node
57 into down state as well as instances running on it.
58 * Nova updates the compute node state and state of the effected instances to
61 Currently nova-compute state will be changing "down", but it takes a long
62 time. Server state keeps as "vm_state: active" and "power_state:
63 running", which is not correct. By having external tool to detect host faults
64 fast, fence host by powering down and then report host down to OpenStack, all
65 these states would reflect to actual situation. Also if OpenStack will not
66 implement automatic actions for fault correlation, external tool can do that.
67 This could be configured for example in server instance METADATA easily and be
68 read by external tool.
72 Liberty priorities have not yet been defined.
76 There needs to be a new API for Admin to state host is down. This API is used
77 to mark compute node and instances running on it down to reflect the real
80 Example on compute node is:
82 * When compute node is up and running:
83 vm_state: active and power_state: running
84 nova-compute state: up status: enabled
85 * When compute node goes down and new API is called to state host is down:
86 vm_state: stopped power_state: shutdown
87 nova-compute state: down status: enabled
89 vm_state values: soft-delete, deleted, resized and error
90 should not be touched.
91 task_state effect needs to be worked out if needs to be touched.
95 There is no attractive alternatives to detect all different host faults than
96 to have a external tool to detect different host faults. For this kind of tool
97 to exist there needs to be new API in Nova to report fault. Currently there
98 must have been some kind of workarounds implemented as cannot trust or get the
99 states from OpenStack fast enough.
107 * Update CLI to report host is down
109 nova host-update command
111 usage: nova host-update [--status <enable|disable>]
112 [--maintenance <enable|disable>]
116 Update host settings.
125 --status <enable|disable>
126 Either enable or disable a host.
128 --maintenance <enable|disable>
129 Either put or resume host to/from maintenance.
132 Report host down to update instance and compute node state in db.
134 * Update Compute API to report host is down:
136 /v2.1/{tenant_id}/os-hosts/{host_name}
138 Normal response codes: 200
141 Parameter Style Type Description
142 host_name URI xsd:string The name of the host of interest to you.
147 "maintenance_mode": "enable"
148 "host_down_reported": "true"
156 "host": "65c5d5b7e3bd44308e67fc50f362aee6",
157 "maintenance_mode": "enabled",
159 "host_down_reported": "true"
165 * New method to nova.compute.api module HostAPI class to have a
166 to mark host related instances and compute node down:
167 set_host_down(context, host_name)
169 * class novaclient.v2.hosts.HostManager(api) method update(host, values)
170 Needs to handle reporting host down.
172 * Schema does not need changes as in db only service and server states are to
177 API call needs admin privileges (in the default policy configuration).
183 Other end user impact
184 ---------------------
189 Only impact is that user can get information faster about instance and
190 compute node state. This also gives possibility to evacuate faster.
191 No impact that would slow down. Host down should be rare occurrence.
193 Other deployer impact
194 ---------------------
195 Developer can make use of any external tool to detect host fault and report it
206 Primary assignee: Tomi Juvonen
207 Other contributors: Ryota Mibu
221 Test cases that exists for enabling or putting host to maintenance should be
222 altered or similar new cases made test new functionality.
227 New API needs to be documented:
229 * Compute API extensions documentation.
230 http://developer.openstack.org/api-ref-compute-v2.1.html
231 * Nova commands documentation.
232 http://docs.openstack.org/user-guide-admin/content/novaclient_commands.html
233 * Compute command-line client documentation.
234 http://docs.openstack.org/cli-reference/content/novaclient_commands.html
235 * nova.compute.api documentation.
236 http://docs.openstack.org/developer/nova/api/nova.compute.api.html
237 * High Availability guide might have page to tell external tool could provide
238 ability to provide faster HA as able to update states by new API.
239 http://docs.openstack.org/high-availability-guide/content/index.html
243 * OPNFV Doctor project: https://wiki.opnfv.org/doctor
244 * OpenStack Instance HA Proposal:
245 http://blog.russellbryant.net/2014/10/15/openstack-instance-ha-proposal/
246 * The Different Facets of OpenStack HA:
247 http://blog.russellbryant.net/2015/03/10/
248 the-different-facets-of-openstack-ha/