+++ /dev/null
-=================================
- Network Configuration Reference
-=================================
-
-Network configuration is critical for building a high performance :term:`Ceph
-Storage Cluster`. The Ceph Storage Cluster does not perform request routing or
-dispatching on behalf of the :term:`Ceph Client`. Instead, Ceph Clients make
-requests directly to Ceph OSD Daemons. Ceph OSD Daemons perform data replication
-on behalf of Ceph Clients, which means replication and other factors impose
-additional loads on Ceph Storage Cluster networks.
-
-Our Quick Start configurations provide a trivial `Ceph configuration file`_ that
-sets monitor IP addresses and daemon host names only. Unless you specify a
-cluster network, Ceph assumes a single "public" network. Ceph functions just
-fine with a public network only, but you may see significant performance
-improvement with a second "cluster" network in a large cluster.
-
-We recommend running a Ceph Storage Cluster with two networks: a public
-(front-side) network and a cluster (back-side) network. To support two networks,
-each :term:`Ceph Node` will need to have more than one NIC. See `Hardware
-Recommendations - Networks`_ for additional details.
-
-.. ditaa::
- +-------------+
- | Ceph Client |
- +----*--*-----+
- | ^
- Request | : Response
- v |
- /----------------------------------*--*-------------------------------------\
- | Public Network |
- \---*--*------------*--*-------------*--*------------*--*------------*--*---/
- ^ ^ ^ ^ ^ ^ ^ ^ ^ ^
- | | | | | | | | | |
- | : | : | : | : | :
- v v v v v v v v v v
- +---*--*---+ +---*--*---+ +---*--*---+ +---*--*---+ +---*--*---+
- | Ceph MON | | Ceph MDS | | Ceph OSD | | Ceph OSD | | Ceph OSD |
- +----------+ +----------+ +---*--*---+ +---*--*---+ +---*--*---+
- ^ ^ ^ ^ ^ ^
- The cluster network relieves | | | | | |
- OSD replication and heartbeat | : | : | :
- traffic from the public network. v v v v v v
- /------------------------------------*--*------------*--*------------*--*---\
- | cCCC Cluster Network |
- \---------------------------------------------------------------------------/
-
-
-There are several reasons to consider operating two separate networks:
-
-#. **Performance:** Ceph OSD Daemons handle data replication for the Ceph
- Clients. When Ceph OSD Daemons replicate data more than once, the network
- load between Ceph OSD Daemons easily dwarfs the network load between Ceph
- Clients and the Ceph Storage Cluster. This can introduce latency and
- create a performance problem. Recovery and rebalancing can
- also introduce significant latency on the public network. See
- `Scalability and High Availability`_ for additional details on how Ceph
- replicates data. See `Monitor / OSD Interaction`_ for details on heartbeat
- traffic.
-
-#. **Security**: While most people are generally civil, a very tiny segment of
- the population likes to engage in what's known as a Denial of Service (DoS)
- attack. When traffic between Ceph OSD Daemons gets disrupted, placement
- groups may no longer reflect an ``active + clean`` state, which may prevent
- users from reading and writing data. A great way to defeat this type of
- attack is to maintain a completely separate cluster network that doesn't
- connect directly to the internet. Also, consider using `Message Signatures`_
- to defeat spoofing attacks.
-
-
-IP Tables
-=========
-
-By default, daemons `bind`_ to ports within the ``6800:7300`` range. You may
-configure this range at your discretion. Before configuring your IP tables,
-check the default ``iptables`` configuration.
-
- sudo iptables -L
-
-Some Linux distributions include rules that reject all inbound requests
-except SSH from all network interfaces. For example::
-
- REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
-
-You will need to delete these rules on both your public and cluster networks
-initially, and replace them with appropriate rules when you are ready to
-harden the ports on your Ceph Nodes.
-
-
-Monitor IP Tables
------------------
-
-Ceph Monitors listen on port ``6789`` by default. Additionally, Ceph Monitors
-always operate on the public network. When you add the rule using the example
-below, make sure you replace ``{iface}`` with the public network interface
-(e.g., ``eth0``, ``eth1``, etc.), ``{ip-address}`` with the IP address of the
-public network and ``{netmask}`` with the netmask for the public network. ::
-
- sudo iptables -A INPUT -i {iface} -p tcp -s {ip-address}/{netmask} --dport 6789 -j ACCEPT
-
-
-MDS IP Tables
--------------
-
-A :term:`Ceph Metadata Server` listens on the first available port on the public
-network beginning at port 6800. Note that this behavior is not deterministic, so
-if you are running more than one OSD or MDS on the same host, or if you restart
-the daemons within a short window of time, the daemons will bind to higher
-ports. You should open the entire 6800-7300 range by default. When you add the
-rule using the example below, make sure you replace ``{iface}`` with the public
-network interface (e.g., ``eth0``, ``eth1``, etc.), ``{ip-address}`` with the IP
-address of the public network and ``{netmask}`` with the netmask of the public
-network.
-
-For example::
-
- sudo iptables -A INPUT -i {iface} -m multiport -p tcp -s {ip-address}/{netmask} --dports 6800:7300 -j ACCEPT
-
-
-OSD IP Tables
--------------
-
-By default, Ceph OSD Daemons `bind`_ to the first available ports on a Ceph Node
-beginning at port 6800. Note that this behavior is not deterministic, so if you
-are running more than one OSD or MDS on the same host, or if you restart the
-daemons within a short window of time, the daemons will bind to higher ports.
-Each Ceph OSD Daemon on a Ceph Node may use up to four ports:
-
-#. One for talking to clients and monitors.
-#. One for sending data to other OSDs.
-#. Two for heartbeating on each interface.
-
-.. ditaa::
- /---------------\
- | OSD |
- | +---+----------------+-----------+
- | | Clients & Monitors | Heartbeat |
- | +---+----------------+-----------+
- | |
- | +---+----------------+-----------+
- | | Data Replication | Heartbeat |
- | +---+----------------+-----------+
- | cCCC |
- \---------------/
-
-When a daemon fails and restarts without letting go of the port, the restarted
-daemon will bind to a new port. You should open the entire 6800-7300 port range
-to handle this possibility.
-
-If you set up separate public and cluster networks, you must add rules for both
-the public network and the cluster network, because clients will connect using
-the public network and other Ceph OSD Daemons will connect using the cluster
-network. When you add the rule using the example below, make sure you replace
-``{iface}`` with the network interface (e.g., ``eth0``, ``eth1``, etc.),
-``{ip-address}`` with the IP address and ``{netmask}`` with the netmask of the
-public or cluster network. For example::
-
- sudo iptables -A INPUT -i {iface} -m multiport -p tcp -s {ip-address}/{netmask} --dports 6800:7300 -j ACCEPT
-
-.. tip:: If you run Ceph Metadata Servers on the same Ceph Node as the
- Ceph OSD Daemons, you can consolidate the public network configuration step.
-
-
-Ceph Networks
-=============
-
-To configure Ceph networks, you must add a network configuration to the
-``[global]`` section of the configuration file. Our 5-minute Quick Start
-provides a trivial `Ceph configuration file`_ that assumes one public network
-with client and server on the same network and subnet. Ceph functions just fine
-with a public network only. However, Ceph allows you to establish much more
-specific criteria, including multiple IP network and subnet masks for your
-public network. You can also establish a separate cluster network to handle OSD
-heartbeat, object replication and recovery traffic. Don't confuse the IP
-addresses you set in your configuration with the public-facing IP addresses
-network clients may use to access your service. Typical internal IP networks are
-often ``192.168.0.0`` or ``10.0.0.0``.
-
-.. tip:: If you specify more than one IP address and subnet mask for
- either the public or the cluster network, the subnets within the network
- must be capable of routing to each other. Additionally, make sure you
- include each IP address/subnet in your IP tables and open ports for them
- as necessary.
-
-.. note:: Ceph uses `CIDR`_ notation for subnets (e.g., ``10.0.0.0/24``).
-
-When you have configured your networks, you may restart your cluster or restart
-each daemon. Ceph daemons bind dynamically, so you do not have to restart the
-entire cluster at once if you change your network configuration.
-
-
-Public Network
---------------
-
-To configure a public network, add the following option to the ``[global]``
-section of your Ceph configuration file.
-
-.. code-block:: ini
-
- [global]
- ...
- public network = {public-network/netmask}
-
-
-Cluster Network
----------------
-
-If you declare a cluster network, OSDs will route heartbeat, object replication
-and recovery traffic over the cluster network. This may improve performance
-compared to using a single network. To configure a cluster network, add the
-following option to the ``[global]`` section of your Ceph configuration file.
-
-.. code-block:: ini
-
- [global]
- ...
- cluster network = {cluster-network/netmask}
-
-We prefer that the cluster network is **NOT** reachable from the public network
-or the Internet for added security.
-
-
-Ceph Daemons
-============
-
-Ceph has one network configuration requirement that applies to all daemons: the
-Ceph configuration file **MUST** specify the ``host`` for each daemon. Ceph also
-requires that a Ceph configuration file specify the monitor IP address and its
-port.
-
-.. important:: Some deployment tools (e.g., ``ceph-deploy``, Chef) may create a
- configuration file for you. **DO NOT** set these values if the deployment
- tool does it for you.
-
-.. tip:: The ``host`` setting is the short name of the host (i.e., not
- an fqdn). It is **NOT** an IP address either. Enter ``hostname -s`` on
- the command line to retrieve the name of the host.
-
-
-.. code-block:: ini
-
- [mon.a]
-
- host = {hostname}
- mon addr = {ip-address}:6789
-
- [osd.0]
- host = {hostname}
-
-
-You do not have to set the host IP address for a daemon. If you have a static IP
-configuration and both public and cluster networks running, the Ceph
-configuration file may specify the IP address of the host for each daemon. To
-set a static IP address for a daemon, the following option(s) should appear in
-the daemon instance sections of your ``ceph.conf`` file.
-
-.. code-block:: ini
-
- [osd.0]
- public addr = {host-public-ip-address}
- cluster addr = {host-cluster-ip-address}
-
-
-.. topic:: One NIC OSD in a Two Network Cluster
-
- Generally, we do not recommend deploying an OSD host with a single NIC in a
- cluster with two networks. However, you may accomplish this by forcing the
- OSD host to operate on the public network by adding a ``public addr`` entry
- to the ``[osd.n]`` section of the Ceph configuration file, where ``n``
- refers to the number of the OSD with one NIC. Additionally, the public
- network and cluster network must be able to route traffic to each other,
- which we don't recommend for security reasons.
-
-
-Network Config Settings
-=======================
-
-Network configuration settings are not required. Ceph assumes a public network
-with all hosts operating on it unless you specifically configure a cluster
-network.
-
-
-Public Network
---------------
-
-The public network configuration allows you specifically define IP addresses
-and subnets for the public network. You may specifically assign static IP
-addresses or override ``public network`` settings using the ``public addr``
-setting for a specific daemon.
-
-``public network``
-
-:Description: The IP address and netmask of the public (front-side) network
- (e.g., ``192.168.0.0/24``). Set in ``[global]``. You may specify
- comma-delimited subnets.
-
-:Type: ``{ip-address}/{netmask} [, {ip-address}/{netmask}]``
-:Required: No
-:Default: N/A
-
-
-``public addr``
-
-:Description: The IP address for the public (front-side) network.
- Set for each daemon.
-
-:Type: IP Address
-:Required: No
-:Default: N/A
-
-
-
-Cluster Network
----------------
-
-The cluster network configuration allows you to declare a cluster network, and
-specifically define IP addresses and subnets for the cluster network. You may
-specifically assign static IP addresses or override ``cluster network``
-settings using the ``cluster addr`` setting for specific OSD daemons.
-
-
-``cluster network``
-
-:Description: The IP address and netmask of the cluster (back-side) network
- (e.g., ``10.0.0.0/24``). Set in ``[global]``. You may specify
- comma-delimited subnets.
-
-:Type: ``{ip-address}/{netmask} [, {ip-address}/{netmask}]``
-:Required: No
-:Default: N/A
-
-
-``cluster addr``
-
-:Description: The IP address for the cluster (back-side) network.
- Set for each daemon.
-
-:Type: Address
-:Required: No
-:Default: N/A
-
-
-Bind
-----
-
-Bind settings set the default port ranges Ceph OSD and MDS daemons use. The
-default range is ``6800:7300``. Ensure that your `IP Tables`_ configuration
-allows you to use the configured port range.
-
-You may also enable Ceph daemons to bind to IPv6 addresses instead of IPv4
-addresses.
-
-
-``ms bind port min``
-
-:Description: The minimum port number to which an OSD or MDS daemon will bind.
-:Type: 32-bit Integer
-:Default: ``6800``
-:Required: No
-
-
-``ms bind port max``
-
-:Description: The maximum port number to which an OSD or MDS daemon will bind.
-:Type: 32-bit Integer
-:Default: ``7300``
-:Required: No.
-
-
-``ms bind ipv6``
-
-:Description: Enables Ceph daemons to bind to IPv6 addresses. Currently the
- messenger *either* uses IPv4 or IPv6, but it cannot do both.
-:Type: Boolean
-:Default: ``false``
-:Required: No
-
-``public bind addr``
-
-:Description: In some dynamic deployments the Ceph MON daemon might bind
- to an IP address locally that is different from the ``public addr``
- advertised to other peers in the network. The environment must ensure
- that routing rules are set correclty. If ``public bind addr`` is set
- the Ceph MON daemon will bind to it locally and use ``public addr``
- in the monmaps to advertise its address to peers. This behavior is limited
- to the MON daemon.
-
-:Type: IP Address
-:Required: No
-:Default: N/A
-
-
-
-Hosts
------
-
-Ceph expects at least one monitor declared in the Ceph configuration file, with
-a ``mon addr`` setting under each declared monitor. Ceph expects a ``host``
-setting under each declared monitor, metadata server and OSD in the Ceph
-configuration file. Optionally, a monitor can be assigned with a priority, and
-the clients will always connect to the monitor with lower value of priority if
-specified.
-
-
-``mon addr``
-
-:Description: A list of ``{hostname}:{port}`` entries that clients can use to
- connect to a Ceph monitor. If not set, Ceph searches ``[mon.*]``
- sections.
-
-:Type: String
-:Required: No
-:Default: N/A
-
-``mon priority``
-
-:Description: The priority of the declared monitor, the lower value the more
- prefered when a client selects a monitor when trying to connect
- to the cluster.
-
-:Type: Unsigned 16-bit Integer
-:Required: No
-:Default: 0
-
-``host``
-
-:Description: The hostname. Use this setting for specific daemon instances
- (e.g., ``[osd.0]``).
-
-:Type: String
-:Required: Yes, for daemon instances.
-:Default: ``localhost``
-
-.. tip:: Do not use ``localhost``. To get your host name, execute
- ``hostname -s`` on your command line and use the name of your host
- (to the first period, not the fully-qualified domain name).
-
-.. important:: You should not specify any value for ``host`` when using a third
- party deployment system that retrieves the host name for you.
-
-
-
-TCP
----
-
-Ceph disables TCP buffering by default.
-
-
-``ms tcp nodelay``
-
-:Description: Ceph enables ``ms tcp nodelay`` so that each request is sent
- immediately (no buffering). Disabling `Nagle's algorithm`_
- increases network traffic, which can introduce latency. If you
- experience large numbers of small packets, you may try
- disabling ``ms tcp nodelay``.
-
-:Type: Boolean
-:Required: No
-:Default: ``true``
-
-
-
-``ms tcp rcvbuf``
-
-:Description: The size of the socket buffer on the receiving end of a network
- connection. Disable by default.
-
-:Type: 32-bit Integer
-:Required: No
-:Default: ``0``
-
-
-
-``ms tcp read timeout``
-
-:Description: If a client or daemon makes a request to another Ceph daemon and
- does not drop an unused connection, the ``ms tcp read timeout``
- defines the connection as idle after the specified number
- of seconds.
-
-:Type: Unsigned 64-bit Integer
-:Required: No
-:Default: ``900`` 15 minutes.
-
-
-
-.. _Scalability and High Availability: ../../../architecture#scalability-and-high-availability
-.. _Hardware Recommendations - Networks: ../../../start/hardware-recommendations#networks
-.. _Ceph configuration file: ../../../start/quick-ceph-deploy/#create-a-cluster
-.. _hardware recommendations: ../../../start/hardware-recommendations
-.. _Monitor / OSD Interaction: ../mon-osd-interaction
-.. _Message Signatures: ../auth-config-ref#signatures
-.. _CIDR: http://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing
-.. _Nagle's Algorithm: http://en.wikipedia.org/wiki/Nagle's_algorithm
Code Review / stor4nfv.git / blobdiff