X-Git-Url: https://gerrit.opnfv.org/gerrit/gitweb?a=blobdiff_plain;f=src%2Fceph%2Fdoc%2Frados%2Fconfiguration%2Fceph-conf.rst;fp=src%2Fceph%2Fdoc%2Frados%2Fconfiguration%2Fceph-conf.rst;h=0000000000000000000000000000000000000000;hb=7da45d65be36d36b880cc55c5036e96c24b53f00;hp=df88452b11eea245f4eefcd4fdb8cdb4ccf76623;hpb=691462d09d0987b47e112d6ee8740375df3c51b2;p=stor4nfv.git diff --git a/src/ceph/doc/rados/configuration/ceph-conf.rst b/src/ceph/doc/rados/configuration/ceph-conf.rst deleted file mode 100644 index df88452..0000000 --- a/src/ceph/doc/rados/configuration/ceph-conf.rst +++ /dev/null @@ -1,629 +0,0 @@ -================== - Configuring Ceph -================== - -When you start the Ceph service, the initialization process activates a series -of daemons that run in the background. A :term:`Ceph Storage Cluster` runs -two types of daemons: - -- :term:`Ceph Monitor` (``ceph-mon``) -- :term:`Ceph OSD Daemon` (``ceph-osd``) - -Ceph Storage Clusters that support the :term:`Ceph Filesystem` run at least one -:term:`Ceph Metadata Server` (``ceph-mds``). Clusters that support :term:`Ceph -Object Storage` run Ceph Gateway daemons (``radosgw``). For your convenience, -each daemon has a series of default values (*i.e.*, many are set by -``ceph/src/common/config_opts.h``). You may override these settings with a Ceph -configuration file. - - -.. _ceph-conf-file: - -The Configuration File -====================== - -When you start a Ceph Storage Cluster, each daemon looks for a Ceph -configuration file (i.e., ``ceph.conf`` by default) that provides the cluster's -configuration settings. For manual deployments, you need to create a Ceph -configuration file. For tools that create configuration files for you (*e.g.*, -``ceph-deploy``, Chef, etc.), you may use the information contained herein as a -reference. The Ceph configuration file defines: - -- Cluster Identity -- Authentication settings -- Cluster membership -- Host names -- Host addresses -- Paths to keyrings -- Paths to journals -- Paths to data -- Other runtime options - -The default Ceph configuration file locations in sequential order include: - -#. ``$CEPH_CONF`` (*i.e.,* the path following the ``$CEPH_CONF`` - environment variable) -#. ``-c path/path`` (*i.e.,* the ``-c`` command line argument) -#. ``/etc/ceph/ceph.conf`` -#. ``~/.ceph/config`` -#. ``./ceph.conf`` (*i.e.,* in the current working directory) - - -The Ceph configuration file uses an *ini* style syntax. You can add comments -by preceding comments with a pound sign (#) or a semi-colon (;). For example: - -.. code-block:: ini - - # <--A number (#) sign precedes a comment. - ; A comment may be anything. - # Comments always follow a semi-colon (;) or a pound (#) on each line. - # The end of the line terminates a comment. - # We recommend that you provide comments in your configuration file(s). - - -.. _ceph-conf-settings: - -Config Sections -=============== - -The configuration file can configure all Ceph daemons in a Ceph Storage Cluster, -or all Ceph daemons of a particular type. To configure a series of daemons, the -settings must be included under the processes that will receive the -configuration as follows: - -``[global]`` - -:Description: Settings under ``[global]`` affect all daemons in a Ceph Storage - Cluster. - -:Example: ``auth supported = cephx`` - -``[osd]`` - -:Description: Settings under ``[osd]`` affect all ``ceph-osd`` daemons in - the Ceph Storage Cluster, and override the same setting in - ``[global]``. - -:Example: ``osd journal size = 1000`` - -``[mon]`` - -:Description: Settings under ``[mon]`` affect all ``ceph-mon`` daemons in - the Ceph Storage Cluster, and override the same setting in - ``[global]``. - -:Example: ``mon addr = 10.0.0.101:6789`` - - -``[mds]`` - -:Description: Settings under ``[mds]`` affect all ``ceph-mds`` daemons in - the Ceph Storage Cluster, and override the same setting in - ``[global]``. - -:Example: ``host = myserver01`` - -``[client]`` - -:Description: Settings under ``[client]`` affect all Ceph Clients - (e.g., mounted Ceph Filesystems, mounted Ceph Block Devices, - etc.). - -:Example: ``log file = /var/log/ceph/radosgw.log`` - - -Global settings affect all instances of all daemon in the Ceph Storage Cluster. -Use the ``[global]`` setting for values that are common for all daemons in the -Ceph Storage Cluster. You can override each ``[global]`` setting by: - -#. Changing the setting in a particular process type - (*e.g.,* ``[osd]``, ``[mon]``, ``[mds]`` ). - -#. Changing the setting in a particular process (*e.g.,* ``[osd.1]`` ). - -Overriding a global setting affects all child processes, except those that -you specifically override in a particular daemon. - -A typical global setting involves activating authentication. For example: - -.. code-block:: ini - - [global] - #Enable authentication between hosts within the cluster. - #v 0.54 and earlier - auth supported = cephx - - #v 0.55 and after - auth cluster required = cephx - auth service required = cephx - auth client required = cephx - - -You can specify settings that apply to a particular type of daemon. When you -specify settings under ``[osd]``, ``[mon]`` or ``[mds]`` without specifying a -particular instance, the setting will apply to all OSDs, monitors or metadata -daemons respectively. - -A typical daemon-wide setting involves setting journal sizes, filestore -settings, etc. For example: - -.. code-block:: ini - - [osd] - osd journal size = 1000 - - -You may specify settings for particular instances of a daemon. You may specify -an instance by entering its type, delimited by a period (.) and by the instance -ID. The instance ID for a Ceph OSD Daemon is always numeric, but it may be -alphanumeric for Ceph Monitors and Ceph Metadata Servers. - -.. code-block:: ini - - [osd.1] - # settings affect osd.1 only. - - [mon.a] - # settings affect mon.a only. - - [mds.b] - # settings affect mds.b only. - - -If the daemon you specify is a Ceph Gateway client, specify the daemon and the -instance, delimited by a period (.). For example:: - - [client.radosgw.instance-name] - # settings affect client.radosgw.instance-name only. - - - -.. _ceph-metavariables: - -Metavariables -============= - -Metavariables simplify Ceph Storage Cluster configuration dramatically. When a -metavariable is set in a configuration value, Ceph expands the metavariable into -a concrete value. Metavariables are very powerful when used within the -``[global]``, ``[osd]``, ``[mon]``, ``[mds]`` or ``[client]`` sections of your -configuration file. Ceph metavariables are similar to Bash shell expansion. - -Ceph supports the following metavariables: - - -``$cluster`` - -:Description: Expands to the Ceph Storage Cluster name. Useful when running - multiple Ceph Storage Clusters on the same hardware. - -:Example: ``/etc/ceph/$cluster.keyring`` -:Default: ``ceph`` - - -``$type`` - -:Description: Expands to one of ``mds``, ``osd``, or ``mon``, depending on the - type of the instant daemon. - -:Example: ``/var/lib/ceph/$type`` - - -``$id`` - -:Description: Expands to the daemon identifier. For ``osd.0``, this would be - ``0``; for ``mds.a``, it would be ``a``. - -:Example: ``/var/lib/ceph/$type/$cluster-$id`` - - -``$host`` - -:Description: Expands to the host name of the instant daemon. - - -``$name`` - -:Description: Expands to ``$type.$id``. -:Example: ``/var/run/ceph/$cluster-$name.asok`` - -``$pid`` - -:Description: Expands to daemon pid. -:Example: ``/var/run/ceph/$cluster-$name-$pid.asok`` - - -.. _ceph-conf-common-settings: - -Common Settings -=============== - -The `Hardware Recommendations`_ section provides some hardware guidelines for -configuring a Ceph Storage Cluster. It is possible for a single :term:`Ceph -Node` to run multiple daemons. For example, a single node with multiple drives -may run one ``ceph-osd`` for each drive. Ideally, you will have a node for a -particular type of process. For example, some nodes may run ``ceph-osd`` -daemons, other nodes may run ``ceph-mds`` daemons, and still other nodes may -run ``ceph-mon`` daemons. - -Each node has a name identified by the ``host`` setting. Monitors also specify -a network address and port (i.e., domain name or IP address) identified by the -``addr`` setting. A basic configuration file will typically specify only -minimal settings for each instance of monitor daemons. For example: - -.. code-block:: ini - - [global] - mon_initial_members = ceph1 - mon_host = 10.0.0.1 - - -.. important:: The ``host`` setting is the short name of the node (i.e., not - an fqdn). It is **NOT** an IP address either. Enter ``hostname -s`` on - the command line to retrieve the name of the node. Do not use ``host`` - settings for anything other than initial monitors unless you are deploying - Ceph manually. You **MUST NOT** specify ``host`` under individual daemons - when using deployment tools like ``chef`` or ``ceph-deploy``, as those tools - will enter the appropriate values for you in the cluster map. - - -.. _ceph-network-config: - -Networks -======== - -See the `Network Configuration Reference`_ for a detailed discussion about -configuring a network for use with Ceph. - - -Monitors -======== - -Ceph production clusters typically deploy with a minimum 3 :term:`Ceph Monitor` -daemons to ensure high availability should a monitor instance crash. At least -three (3) monitors ensures that the Paxos algorithm can determine which version -of the :term:`Ceph Cluster Map` is the most recent from a majority of Ceph -Monitors in the quorum. - -.. note:: You may deploy Ceph with a single monitor, but if the instance fails, - the lack of other monitors may interrupt data service availability. - -Ceph Monitors typically listen on port ``6789``. For example: - -.. code-block:: ini - - [mon.a] - host = hostName - mon addr = 150.140.130.120:6789 - -By default, Ceph expects that you will store a monitor's data under the -following path:: - - /var/lib/ceph/mon/$cluster-$id - -You or a deployment tool (e.g., ``ceph-deploy``) must create the corresponding -directory. With metavariables fully expressed and a cluster named "ceph", the -foregoing directory would evaluate to:: - - /var/lib/ceph/mon/ceph-a - -For additional details, see the `Monitor Config Reference`_. - -.. _Monitor Config Reference: ../mon-config-ref - - -.. _ceph-osd-config: - - -Authentication -============== - -.. versionadded:: Bobtail 0.56 - -For Bobtail (v 0.56) and beyond, you should expressly enable or disable -authentication in the ``[global]`` section of your Ceph configuration file. :: - - auth cluster required = cephx - auth service required = cephx - auth client required = cephx - -Additionally, you should enable message signing. See `Cephx Config Reference`_ for details. - -.. important:: When upgrading, we recommend expressly disabling authentication - first, then perform the upgrade. Once the upgrade is complete, re-enable - authentication. - -.. _Cephx Config Reference: ../auth-config-ref - - -.. _ceph-monitor-config: - - -OSDs -==== - -Ceph production clusters typically deploy :term:`Ceph OSD Daemons` where one node -has one OSD daemon running a filestore on one storage drive. A typical -deployment specifies a journal size. For example: - -.. code-block:: ini - - [osd] - osd journal size = 10000 - - [osd.0] - host = {hostname} #manual deployments only. - - -By default, Ceph expects that you will store a Ceph OSD Daemon's data with the -following path:: - - /var/lib/ceph/osd/$cluster-$id - -You or a deployment tool (e.g., ``ceph-deploy``) must create the corresponding -directory. With metavariables fully expressed and a cluster named "ceph", the -foregoing directory would evaluate to:: - - /var/lib/ceph/osd/ceph-0 - -You may override this path using the ``osd data`` setting. We don't recommend -changing the default location. Create the default directory on your OSD host. - -:: - - ssh {osd-host} - sudo mkdir /var/lib/ceph/osd/ceph-{osd-number} - -The ``osd data`` path ideally leads to a mount point with a hard disk that is -separate from the hard disk storing and running the operating system and -daemons. If the OSD is for a disk other than the OS disk, prepare it for -use with Ceph, and mount it to the directory you just created:: - - ssh {new-osd-host} - sudo mkfs -t {fstype} /dev/{disk} - sudo mount -o user_xattr /dev/{hdd} /var/lib/ceph/osd/ceph-{osd-number} - -We recommend using the ``xfs`` file system when running -:command:`mkfs`. (``btrfs`` and ``ext4`` are not recommended and no -longer tested.) - -See the `OSD Config Reference`_ for additional configuration details. - - -Heartbeats -========== - -During runtime operations, Ceph OSD Daemons check up on other Ceph OSD Daemons -and report their findings to the Ceph Monitor. You do not have to provide any -settings. However, if you have network latency issues, you may wish to modify -the settings. - -See `Configuring Monitor/OSD Interaction`_ for additional details. - - -.. _ceph-logging-and-debugging: - -Logs / Debugging -================ - -Sometimes you may encounter issues with Ceph that require -modifying logging output and using Ceph's debugging. See `Debugging and -Logging`_ for details on log rotation. - -.. _Debugging and Logging: ../../troubleshooting/log-and-debug - - -Example ceph.conf -================= - -.. literalinclude:: demo-ceph.conf - :language: ini - -.. _ceph-runtime-config: - -Runtime Changes -=============== - -Ceph allows you to make changes to the configuration of a ``ceph-osd``, -``ceph-mon``, or ``ceph-mds`` daemon at runtime. This capability is quite -useful for increasing/decreasing logging output, enabling/disabling debug -settings, and even for runtime optimization. The following reflects runtime -configuration usage:: - - ceph tell {daemon-type}.{id or *} injectargs --{name} {value} [--{name} {value}] - -Replace ``{daemon-type}`` with one of ``osd``, ``mon`` or ``mds``. You may apply -the runtime setting to all daemons of a particular type with ``*``, or specify -a specific daemon's ID (i.e., its number or letter). For example, to increase -debug logging for a ``ceph-osd`` daemon named ``osd.0``, execute the following:: - - ceph tell osd.0 injectargs --debug-osd 20 --debug-ms 1 - -In your ``ceph.conf`` file, you may use spaces when specifying a -setting name. When specifying a setting name on the command line, -ensure that you use an underscore or hyphen (``_`` or ``-``) between -terms (e.g., ``debug osd`` becomes ``--debug-osd``). - - -Viewing a Configuration at Runtime -================================== - -If your Ceph Storage Cluster is running, and you would like to see the -configuration settings from a running daemon, execute the following:: - - ceph daemon {daemon-type}.{id} config show | less - -If you are on a machine where osd.0 is running, the command would be:: - - ceph daemon osd.0 config show | less - -Reading Configuration Metadata at Runtime -========================================= - -Information about the available configuration options is available via -the ``config help`` command: - -:: - - ceph daemon {daemon-type}.{id} config help | less - - -This metadata is primarily intended to be used when integrating other -software with Ceph, such as graphical user interfaces. The output is -a list of JSON objects, for example: - -:: - - { - "name": "mon_host", - "type": "std::string", - "level": "basic", - "desc": "list of hosts or addresses to search for a monitor", - "long_desc": "This is a comma, whitespace, or semicolon separated list of IP addresses or hostnames. Hostnames are resolved via DNS and all A or AAAA records are included in the search list.", - "default": "", - "daemon_default": "", - "tags": [], - "services": [ - "common" - ], - "see_also": [], - "enum_values": [], - "min": "", - "max": "" - } - -type -____ - -The type of the setting, given as a C++ type name. - -level -_____ - -One of `basic`, `advanced`, `dev`. The `dev` options are not intended -for use outside of development and testing. - -desc -____ - -A short description -- this is a sentence fragment suitable for display -in small spaces like a single line in a list. - -long_desc -_________ - -A full description of what the setting does, this may be as long as needed. - -default -_______ - -The default value, if any. - -daemon_default -______________ - -An alternative default used for daemons (services) as opposed to clients. - -tags -____ - -A list of strings indicating topics to which this setting relates. Examples -of tags are `performance` and `networking`. - -services -________ - -A list of strings indicating which Ceph services the setting relates to, such -as `osd`, `mds`, `mon`. For settings that are relevant to any Ceph client -or server, `common` is used. - -see_also -________ - -A list of strings indicating other configuration options that may also -be of interest to a user setting this option. - -enum_values -___________ - -Optional: a list of strings indicating the valid settings. - -min, max -________ - -Optional: upper and lower (inclusive) bounds on valid settings. - - - - -Running Multiple Clusters -========================= - -With Ceph, you can run multiple Ceph Storage Clusters on the same hardware. -Running multiple clusters provides a higher level of isolation compared to -using different pools on the same cluster with different CRUSH rulesets. A -separate cluster will have separate monitor, OSD and metadata server processes. -When running Ceph with default settings, the default cluster name is ``ceph``, -which means you would save your Ceph configuration file with the file name -``ceph.conf`` in the ``/etc/ceph`` default directory. - -See `ceph-deploy new`_ for details. -.. _ceph-deploy new:../ceph-deploy-new - -When you run multiple clusters, you must name your cluster and save the Ceph -configuration file with the name of the cluster. For example, a cluster named -``openstack`` will have a Ceph configuration file with the file name -``openstack.conf`` in the ``/etc/ceph`` default directory. - -.. important:: Cluster names must consist of letters a-z and digits 0-9 only. - -Separate clusters imply separate data disks and journals, which are not shared -between clusters. Referring to `Metavariables`_, the ``$cluster`` metavariable -evaluates to the cluster name (i.e., ``openstack`` in the foregoing example). -Various settings use the ``$cluster`` metavariable, including: - -- ``keyring`` -- ``admin socket`` -- ``log file`` -- ``pid file`` -- ``mon data`` -- ``mon cluster log file`` -- ``osd data`` -- ``osd journal`` -- ``mds data`` -- ``rgw data`` - -See `General Settings`_, `OSD Settings`_, `Monitor Settings`_, `MDS Settings`_, -`RGW Settings`_ and `Log Settings`_ for relevant path defaults that use the -``$cluster`` metavariable. - -.. _General Settings: ../general-config-ref -.. _OSD Settings: ../osd-config-ref -.. _Monitor Settings: ../mon-config-ref -.. _MDS Settings: ../../../cephfs/mds-config-ref -.. _RGW Settings: ../../../radosgw/config-ref/ -.. _Log Settings: ../../troubleshooting/log-and-debug - - -When creating default directories or files, you should use the cluster -name at the appropriate places in the path. For example:: - - sudo mkdir /var/lib/ceph/osd/openstack-0 - sudo mkdir /var/lib/ceph/mon/openstack-a - -.. important:: When running monitors on the same host, you should use - different ports. By default, monitors use port 6789. If you already - have monitors using port 6789, use a different port for your other cluster(s). - -To invoke a cluster other than the default ``ceph`` cluster, use the -``-c {filename}.conf`` option with the ``ceph`` command. For example:: - - ceph -c {cluster-name}.conf health - ceph -c openstack.conf health - - -.. _Hardware Recommendations: ../../../start/hardware-recommendations -.. _Network Configuration Reference: ../network-config-ref -.. _OSD Config Reference: ../osd-config-ref -.. _Configuring Monitor/OSD Interaction: ../mon-osd-interaction -.. _ceph-deploy new: ../../deployment/ceph-deploy-new#naming-a-cluster