5 Ceph is used to build a storage system accross all machines
8 We consider the following architecture
10 TODO: add schema (4 machines: ceph-admin, 3 ceph-nodes: opensteak9{2,3,4})
14 192.168.0.0/24 is the cluster network (use for storage)
15 192.168.1.0/24 is the management network (use for admin task)
19 ## Ceph-admin machine preparation
21 This is done on an Ubuntu 14.04 64b server
23 ### Install ceph-deploy
25 wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' | sudo apt-key add -
26 echo deb http://ceph.com/debian-firefly/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
27 sudo apt-get update && sudo apt-get install ceph-deploy
32 sudo apt-get install ntp
33 sudo service ntp restart
36 ### Create a ceph user on each node (ceph-admin included)
38 sudo useradd -d /home/ceph -m ceph
44 echo "ceph ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph
45 sudo chmod 0440 /etc/sudoers.d/ceph
48 * *Note: if you think this can be a security treat, remove the ceph user from sudoers after installation is complete*
50 * *Note 2: the ceph documentation ask for this user: http://ceph.com/docs/master/rados/deployment/preflight-checklist/?highlight=sudoers*
53 ### Add each node in hosts file (ceph-admin included)
55 sudo bash -c ' cat << EOF >> /etc/hosts
56 192.168.1.200 ceph-admin
57 192.168.1.92 opensteak92
58 192.168.1.93 opensteak93
59 192.168.1.94 opensteak94
63 ### Create and copy a passwordless ssh key to each node
66 ssh-copy-id ceph@ceph-admin
67 ssh-copy-id ceph@opensteak92
68 ssh-copy-id ceph@opensteak93
69 ssh-copy-id ceph@opensteak94
72 ### Create a .ssh/config file to connect automatically
74 cat << EOF >> .ssh/config
90 ## Ceph storage cluster
91 All these commands must be run inside the ceph-admin machine as a regular user
99 ### Deploy initial monitor on first node
101 ceph-deploy new opensteak92
105 We set default pool size to 2 and public/cluster networks:
108 cat << EOF >> ceph.conf
109 osd pool default size = 2
110 public network = 192.168.1.0/24
111 cluster network = 192.168.0.0/24
115 ### Install ceph in all nodes
117 ceph-deploy --username ceph install ceph-admin opensteak92 opensteak93 opensteak94
120 ### Create initial monitor and gather the keys
122 ceph-deploy --username ceph mon create-initial
125 ### Create and add OSD
126 We will use hard disk (/dev/sdb) for storage: http://docs.ceph.com/docs/master/rados/deployment/ceph-deploy-osd/
129 ceph-deploy --username ceph osd create opensteak93:sdb
130 ceph-deploy --username ceph osd create opensteak94:sdb
133 ### Prepare all nodes to administer the cluster
134 Prepare all nodes with a ceph.conf and ceph.client.admin.keyring keyring so that it can administer the cluster:
137 ceph-deploy admin ceph-admin opensteak92 opensteak93 opensteak94
138 sudo chmod +r /etc/ceph/ceph.client.admin.keyring
141 ### Add a metadata server in first node
143 ceph-deploy--username ceph mds create opensteak92
147 ### Extend the OSD pool
148 We decided to extend OSD pool by adding the first node as well:
151 ceph-deploy --username ceph osd create opensteak92:sdb
154 ### Extend the monitors
155 In the same spirit, extend the monitor by adding the two last nodes and check the status
157 ceph-deploy --username ceph mon create opensteak93 opensteak94
158 ceph quorum_status --format json-pretty
166 ## Create a file system
172 I you don't have data and metadata pools, create it:
174 ceph osd pool create cephfs_data 64
175 ceph osd pool create cephfs_metadata 64
178 Then enable filesystem on the cephfs_data pool:
180 ceph fs new cephfs cephfs_metadata cephfs_data
190 0 rbd,1 cephfs_data,2 cephfs_metadata,
193 You can check as well with:
196 name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]
199 e5: 1/1/1 up {0=opensteak92=up:active}
203 For each node you want to mount ceph in **/mnt/cephfs/**, run:
205 ssh opensteak9x "cat /etc/ceph/ceph.client.admin.keyring |grep key|awk '{print \$3}'|sudo tee /etc/ceph/ceph.client.admin.key"
207 ssh opensteak9x "sudo mkdir /mnt/cephfs"
209 ssh opensteak9x "echo '192.168.1.92:6789:/ /mnt/cephfs ceph name=admin,secretfile=/etc/ceph/ceph.client.admin.key,noatime 0 2' | sudo tee --append /etc/fstab && sudo mount /mnt/cephfs"
212 This will add a line in fstab so the file system will automatically be mounted on boot.
216 * create a python/bash script that will install & check that the cluster is well configured (do all of this automatically)
217 * create a conf file that will be used by the above script to describe the architecture?