+++ /dev/null
-# author: Min chen(minchen@ubuntukylin.com) 2014 2015
-
-------------- ceph rbd recover tool -------------
-
- ceph rbd recover tool is used for recovering ceph rbd image, when all ceph services are killed.
-it is based on ceph-0.80.x (Firefly and newer)
- currently, ceph service(ceph-mon, ceph-osd) evently are not avaiable caused by bugs or sth else
-, especially on large scale ceph cluster, so that the ceph cluster can not supply service
-and rbd images can not be accessed. In this case, a tool to recover rbd image is nessecary.
- ceph rbd recover tool is just used for this, it can collect all objects of an image from distributed
-osd nodes with the latest pg epoch, and splice objects by offset to a complete image. To make sure
-object data is complete, this tool does flush osd journal on each osd node before recovering.
- but, there are some limitions:
--need ssh service and unobstructed network
--osd data must be accessed on local disk
--clone image is not supported, while snapshot is supported
--only support relicated pool
-
-before you run this tool, you should make sure that:
-1). all processes (ceph-osd, ceph-mon, ceph-mds) are shutdown
-2). ssh deamon is running & network is ok (ssh to each node without password)
-3). ceph-kvstore-tool is installed(for ubuntu: apt-get install ceph-test)
-4). osd disk is not crashed and data can be accessed on local filesystem
-
--architecture:
-
- +---- osd.0
- |
-admin_node -----------+---- osd.1
- |
- +---- osd.2
- |
- ......
-
--files:
-admin_node: {rbd-recover-tool common_h epoch_h metadata_h database_h}
-osd: {osd_job common_h epoch_h metadata_h} #/var/rbd_tool/osd_job
-in this architecture, admin_node acts as client, osds act as server.
-so, they run diffrent files:
-on admin_node run: rbd-recover-tool <action> [<parameters>]
-on osd node run: ./osd_job <funtion> <parameters>
-admin_node will copy files: osd_job, common_h, epoch_h, metadata_h to remote osd node
-
-
--config file
-before you run this tool, make sure write config files first
-osd_host_path: osd hostnames and osd data path #user input
- osdhost0 /var/lib/ceph/osd/ceph-0
- osdhost1 /var/lib/ceph/osd/ceph-1
- ......
-mon_host: all mon node hostname #user input
- monhost0
- monhost1
- ......
-mds_host: all mds node hostname #user input
- mdshost0
- mdshost1
- ......
-then, init_env_admin function will create file: osd_host
-osd_host: all osd node hostname #generated by admin_job, user ignore it
- osdhost0
- osdhost1
- ......
-
-
--usage:
-rbd-recovert-tool <operation>
-<operation> :
-database #generating offline database: hobject path, node hostname, pg_epoch and image metadata
-list #list all images from offline database
-lookup <pool_id>/<image_name>[@[<snap_name>]] #lookup image metadata in offline database
-recover <pool_id><image_name>[@[<snap_name>]] [/path/to/store/image] #recover image data according to image metadata
-
--steps:
-1. stop all ceph services: ceph-mon, ceph-osd, ceph-mds
-2. setup config files: osd_host_path, mon_host, mds_host
-3. rbd-recover-tool database # wait a long time
-4. rbd-recover-tool list
-4. rbd-recover-tool recover <pool_id>/<image_name>[@[<image_name>]] [/path/to/store/image]
-
-
--debug & error check
-if admin_node operation is failed, you can check it on osd node
-cd /var/rbd_tool/osd_job
-./osd_job <operation>
-<opeartion> :
-do_image_id <image_id_hobject> #get image id of image format v2
-do_image_id <image_header_hobject> #get image id of image format v1
-do_image_metadata_v1 <image_header_hobject> #get image metadata of image format v1, maybe pg epoch is not latest
-do_image_metadata_v2 <image_header_hobject> #get image metadata of image format v2, maybe pg epoch is not latest
-do_image_list #get all images on this osd(image head hobject)
-do_pg_epoch #get all pg epoch and store it in /var/rbd_tool/single_node/node_pg_epoch
-do_omap_list #list all omap headers and omap entries on this osd
-
-
--FAQ
-file FAQ lists some common confusing cases while testing
Code Review / stor4nfv.git / blobdiff