1 # author: Min chen(minchen@ubuntukylin.com) 2014 2015
3 ------------- ceph rbd recover tool -------------
5 ceph rbd recover tool is used for recovering ceph rbd image, when all ceph services are killed.
6 it is based on ceph-0.80.x (Firefly and newer)
7 currently, ceph service(ceph-mon, ceph-osd) evently are not avaiable caused by bugs or sth else
8 , especially on large scale ceph cluster, so that the ceph cluster can not supply service
9 and rbd images can not be accessed. In this case, a tool to recover rbd image is nessecary.
10 ceph rbd recover tool is just used for this, it can collect all objects of an image from distributed
11 osd nodes with the latest pg epoch, and splice objects by offset to a complete image. To make sure
12 object data is complete, this tool does flush osd journal on each osd node before recovering.
13 but, there are some limitions:
14 -need ssh service and unobstructed network
15 -osd data must be accessed on local disk
16 -clone image is not supported, while snapshot is supported
17 -only support relicated pool
19 before you run this tool, you should make sure that:
20 1). all processes (ceph-osd, ceph-mon, ceph-mds) are shutdown
21 2). ssh deamon is running & network is ok (ssh to each node without password)
22 3). ceph-kvstore-tool is installed(for ubuntu: apt-get install ceph-test)
23 4). osd disk is not crashed and data can be accessed on local filesystem
29 admin_node -----------+---- osd.1
36 admin_node: {rbd-recover-tool common_h epoch_h metadata_h database_h}
37 osd: {osd_job common_h epoch_h metadata_h} #/var/rbd_tool/osd_job
38 in this architecture, admin_node acts as client, osds act as server.
39 so, they run diffrent files:
40 on admin_node run: rbd-recover-tool <action> [<parameters>]
41 on osd node run: ./osd_job <funtion> <parameters>
42 admin_node will copy files: osd_job, common_h, epoch_h, metadata_h to remote osd node
46 before you run this tool, make sure write config files first
47 osd_host_path: osd hostnames and osd data path #user input
48 osdhost0 /var/lib/ceph/osd/ceph-0
49 osdhost1 /var/lib/ceph/osd/ceph-1
51 mon_host: all mon node hostname #user input
55 mds_host: all mds node hostname #user input
59 then, init_env_admin function will create file: osd_host
60 osd_host: all osd node hostname #generated by admin_job, user ignore it
67 rbd-recovert-tool <operation>
69 database #generating offline database: hobject path, node hostname, pg_epoch and image metadata
70 list #list all images from offline database
71 lookup <pool_id>/<image_name>[@[<snap_name>]] #lookup image metadata in offline database
72 recover <pool_id><image_name>[@[<snap_name>]] [/path/to/store/image] #recover image data according to image metadata
75 1. stop all ceph services: ceph-mon, ceph-osd, ceph-mds
76 2. setup config files: osd_host_path, mon_host, mds_host
77 3. rbd-recover-tool database # wait a long time
78 4. rbd-recover-tool list
79 4. rbd-recover-tool recover <pool_id>/<image_name>[@[<image_name>]] [/path/to/store/image]
83 if admin_node operation is failed, you can check it on osd node
84 cd /var/rbd_tool/osd_job
87 do_image_id <image_id_hobject> #get image id of image format v2
88 do_image_id <image_header_hobject> #get image id of image format v1
89 do_image_metadata_v1 <image_header_hobject> #get image metadata of image format v1, maybe pg epoch is not latest
90 do_image_metadata_v2 <image_header_hobject> #get image metadata of image format v2, maybe pg epoch is not latest
91 do_image_list #get all images on this osd(image head hobject)
92 do_pg_epoch #get all pg epoch and store it in /var/rbd_tool/single_node/node_pg_epoch
93 do_omap_list #list all omap headers and omap entries on this osd
97 file FAQ lists some common confusing cases while testing