1 .. This work is licensed under a Creative Commons Attribution 4.0 International
3 .. http://creativecommons.org/licenses/by/4.0
4 .. (c) OPNFV, Yin Kanglin and others.
5 .. 14_ykl@tongji.edu.cn
7 *************************************
8 Yardstick Test Case Description TC057
9 *************************************
11 +-----------------------------------------------------------------------------+
12 |OpenStack Controller Cluster Management Service High Availability |
14 +--------------+--------------------------------------------------------------+
15 |test case id | OPNFV_YARDSTICK_TC057_HA: OpenStack Controller Cluster |
16 | | Management Service High Availability |
18 +--------------+--------------------------------------------------------------+
19 |test purpose | This test case will verify the quorum configuration of the |
20 | | cluster manager(pacemaker) on controller nodes. When a |
21 | | controller node , which holds all active application |
22 | | resources, failed to communicate with other cluster nodes |
23 | | (via corosync), the test case will check whether the standby |
24 | | application resources will take place of those active |
25 | | application resources which should be regarded to be down in |
26 | | the cluster manager. |
27 +--------------+--------------------------------------------------------------+
28 |test method | This test case kills the processes of cluster messaging |
29 | | service(corosync) on a selected controller node(the node |
30 | | holds the active application resources), then checks whether |
31 | | active application resources are switched to other |
32 | | controller nodes and whether the Openstack commands are OK. |
33 +--------------+--------------------------------------------------------------+
34 |attackers | In this test case, an attacker called "kill-process" is |
35 | | needed. This attacker includes three parameters: |
36 | | 1) fault_type: which is used for finding the attacker's |
37 | | scripts. It should be always set to "kill-process" in this |
39 | | 2) process_name: which is the process name of the load |
40 | | balance service. If there are multiple processes use the |
41 | | same name on the host, all of them are killed by this |
43 | | 3) host: which is the name of a control node being attacked. |
45 | | In this case, this process name should set to "corosync" , |
47 | | -fault_type: "kill-process" |
48 | | -process_name: "corosync" |
50 +--------------+--------------------------------------------------------------+
51 |monitors | In this test case, a kind of monitor is needed: |
52 | | 1. the "openstack-cmd" monitor constantly request a specific |
53 | | Openstack command, which needs two parameters: |
54 | | 1) monitor_type: which is used for finding the monitor class |
55 | | and related scripts. It should be always set to |
56 | | "openstack-cmd" for this monitor. |
57 | | 2) command_name: which is the command name used for request |
59 | | In this case, the command_name of monitor1 should be |
60 | | services that are managed by the cluster manager. |
61 | | (Since rabbitmq and haproxy are managed by pacemaker, |
62 | | most Openstack Services can be used to check high |
63 | | availability in this case) |
67 | | -monitor_type: "openstack-cmd" |
68 | | -command_name: "nova image-list" |
70 | | -monitor_type: "openstack-cmd" |
71 | | -command_name: "neutron router-list" |
73 | | -monitor_type: "openstack-cmd" |
74 | | -command_name: "heat stack-list" |
76 | | -monitor_type: "openstack-cmd" |
77 | | -command_name: "cinder list" |
79 +--------------+--------------------------------------------------------------+
80 |checkers | In this test case, a checker is needed, the checker will |
81 | | the status of application resources in pacemaker and the |
82 | | checker have three parameters: |
83 | | 1) checker_type: which is used for finding the result |
84 | | checker class and related scripts. In this case the checker |
85 | | type will be "pacemaker-check-resource" |
86 | | 2) resource_name: the application resource name |
87 | | 3) resource_status: the expected status of the resource |
88 | | 4) expectedValue: the expected value for the output of the |
89 | | checker script, in the case the expected value will be the |
90 | | identifier in the cluster manager |
91 | | 3) condition: whether the expected value is in the output of |
92 | | checker script or is totally same with the output. |
93 | | (note: pcs is required to installed on controller node in |
94 | | order to run this checker) |
98 | | -checker_type: "pacemaker-check-resource" |
99 | | -resource_name: "p_rabbitmq-server" |
100 | | -resource_status: "Stopped" |
101 | | -expectedValue: "node-1" |
102 | | -condition: "in" |
104 | | -checker_type: "pacemaker-check-resource" |
105 | | -resource_name: "p_rabbitmq-server" |
106 | | -resource_status: "Master" |
107 | | -expectedValue: "node-2" |
108 | | -condition: "in" |
109 +--------------+--------------------------------------------------------------+
110 |metrics | In this test case, there are two metrics: |
111 | | 1)service_outage_time: which indicates the maximum outage |
112 | | time (seconds) of the specified Openstack command request. |
113 +--------------+--------------------------------------------------------------+
114 |test tool | None. Self-developed. |
115 +--------------+--------------------------------------------------------------+
116 |references | ETSI NFV REL001 |
117 +--------------+--------------------------------------------------------------+
118 |configuration | This test case needs two configuration files: |
119 | | 1) test case file: opnfv_yardstick_tc057.yaml |
120 | | -Attackers: see above "attackers" description |
121 | | -Monitors: see above "monitors" description |
122 | | -Checkers: see above "checkers" description |
123 | | -Steps: the test case execution step, see "test sequence" |
124 | | description below |
126 | | 2)POD file: pod.yaml |
127 | | The POD configuration should record on pod.yaml first. |
128 | | the "host" item in this test case will use the node name in |
130 +--------------+------+----------------------------------+--------------------+
131 |test sequence | description and expected result |
133 +--------------+--------------------------------------------------------------+
134 |step 1 | start monitors: |
135 | | each monitor will run with independently process |
137 | | Result: The monitor info will be collected. |
139 +--------------+--------------------------------------------------------------+
140 |step 2 | do attacker: connect the host through SSH, and then execute |
141 | | the kill process script with param value specified by |
144 | | Result: Process will be killed. |
146 +--------------+--------------------------------------------------------------+
147 |step 3 | do checker: check whether the status of application |
148 | | resources on different nodes are updated |
150 +--------------+--------------------------------------------------------------+
151 |step 4 | stop monitors after a period of time specified by |
154 | | Result: The monitor info will be aggregated. |
156 +--------------+--------------------------------------------------------------+
157 |step 5 | verify the SLA |
159 | | Result: The test case is passed or not. |
161 +--------------+------+----------------------------------+--------------------+
162 |post-action | It is the action when the test cases exist. It will check |
163 | | the status of the cluster messaging process(corosync) on the |
164 | | host, and restart the process if it is not running for next |
166 | | Notice: This post-action uses 'lsb_release' command to check |
167 | | the host linux distribution and determine the OpenStack |
168 | | service name to restart the process. Lack of 'lsb_release' |
169 | | on the host may cause failure to restart the process. |
171 +--------------+------+----------------------------------+--------------------+
172 |test verdict | Fails only if SLA is not passed, or if there is a test case |
173 | | execution problem. |
175 +--------------+--------------------------------------------------------------+
Code Review / yardstick.git / blob