+++ /dev/null
-msgr2 protocol
-==============
-
-This is a revision of the legacy Ceph on-wire protocol that was
-implemented by the SimpleMessenger. It addresses performance and
-security issues.
-
-Definitions
------------
-
-* *client* (C): the party initiating a (TCP) connection
-* *server* (S): the party accepting a (TCP) connection
-* *connection*: an instance of a (TCP) connection between two processes.
-* *entity*: a ceph entity instantiation, e.g. 'osd.0'. each entity
- has one or more unique entity_addr_t's by virtue of the 'nonce'
- field, which is typically a pid or random value.
-* *stream*: an exchange, passed over a connection, between two unique
- entities. in the future multiple entities may coexist within the
- same process.
-* *session*: a stateful session between two entities in which message
- exchange is ordered and lossless. A session might span multiple
- connections (and streams) if there is an interruption (TCP connection
- disconnect).
-* *frame*: a discrete message sent between the peers. Each frame
- consists of a tag (type code), stream id, payload, and (if signing
- or encryption is enabled) some other fields. See below for the
- structure.
-* *stream id*: a 32-bit value that uniquely identifies a stream within
- a given connection. the stream id implicitly instantiated when the send
- sends a frame using that id.
-* *tag*: a single-byte type code associated with a frame. The tag
- determines the structure of the payload.
-
-Phases
-------
-
-A connection has two distinct phases:
-
-#. banner
-#. frame exchange for one or more strams
-
-A stream has three distinct phases:
-
-#. authentication
-#. message flow handshake
-#. message exchange
-
-Banner
-------
-
-Both the client and server, upon connecting, send a banner::
-
- "ceph %x %x\n", protocol_features_suppored, protocol_features_required
-
-The protocol features are a new, distinct namespace. Initially no
-features are defined or required, so this will be "ceph 0 0\n".
-
-If the remote party advertises required features we don't support, we
-can disconnect.
-
-Frame format
-------------
-
-All further data sent or received is contained by a frame. Each frame has
-the form::
-
- stream_id (le32)
- frame_len (le32)
- tag (TAG_* byte)
- payload
- [payload padding -- only present after stream auth phase]
- [signature -- only present after stream auth phase]
-
-* frame_len includes everything after the frame_len le32 up to the end of the
- frame (all payloads, signatures, and padding).
-
-* The payload format and length is determined by the tag.
-
-* The signature portion is only present in a given stream if the
- authentication phase has completed (TAG_AUTH_DONE has been sent) and
- signatures are enabled.
-
-
-Authentication
---------------
-
-* TAG_AUTH_METHODS (server only): list authentication methods (none, cephx, ...)::
-
- __le32 num_methods;
- __le32 methods[num_methods]; // CEPH_AUTH_{NONE, CEPHX}
-
-* TAG_AUTH_SET_METHOD (client only): set auth method for this connection::
-
- __le32 method;
-
- - The selected auth method determines the sig_size and block_size in any
- subsequent messages (TAG_AUTH_DONE and non-auth messages).
-
-* TAG_AUTH_BAD_METHOD (server only): reject client-selected auth method::
-
- __le32 method
-
-* TAG_AUTH: client->server or server->client auth message::
-
- __le32 len;
- method specific payload
-
-* TAG_AUTH_DONE::
-
- confounder (block_size bytes of random garbage)
- __le64 flags
- FLAG_ENCRYPTED 1
- FLAG_SIGNED 2
- signature
-
- - The client first says AUTH_DONE, and the server replies to
- acknowledge it.
-
-
-Message frame format
---------------------
-
-The frame format is fixed (see above), but can take three different
-forms, depending on the AUTH_DONE flags:
-
-* If neither FLAG_SIGNED or FLAG_ENCRYPTED is specified, things are simple::
-
- stream_id
- frame_len
- tag
- payload
- payload_padding (out to auth block_size)
-
-* If FLAG_SIGNED has been specified::
-
- stream_id
- frame_len
- tag
- payload
- payload_padding (out to auth block_size)
- signature (sig_size bytes)
-
- Here the padding just makes life easier for the signature. It can be
- random data to add additional confounder. Note also that the
- signature input must include some state from the session key and the
- previous message.
-
-* If FLAG_ENCRYPTED has been specified::
-
- stream_id
- frame_len
- {
- payload_sig_length
- payload
- payload_padding (out to auth block_size)
- } ^ stream cipher
-
- Note that the padding ensures that the total frame is a multiple of
- the auth method's block_size so that the message can be sent out over
- the wire without waiting for the next frame in the stream.
-
-
-Message flow handshake
-----------------------
-
-In this phase the peers identify each other and (if desired) reconnect to
-an established session.
-
-* TAG_IDENT: identify ourselves::
-
- entity_addrvec_t addr(s)
- __u8 my type (CEPH_ENTITY_TYPE_*)
- __le32 protocol version
- __le64 features supported (CEPH_FEATURE_* bitmask)
- __le64 features required (CEPH_FEATURE_* bitmask)
- __le64 flags (CEPH_MSG_CONNECT_* bitmask)
- __le64 cookie (a client identifier, assigned by the sender. unique on the sender.)
-
- - client will send first, server will reply with same.
-
-* TAG_IDENT_MISSING_FEATURES (server only): complain about a TAG_IDENT with too few features::
-
- __le64 features we require that peer didn't advertise
-
-* TAG_IDENT_BAD_PROTOCOL (server only): complain about an old protocol version::
-
- __le32 protocol_version (our protocol version)
-
-* TAG_RECONNECT (client only): reconnect to an established session::
-
- __le64 cookie
- __le64 global_seq
- __le64 connect_seq
- __le64 msg_seq (the last msg seq received)
-
-* TAG_RECONNECT_OK (server only): acknowledge a reconnect attempt::
-
- __le64 msg_seq (last msg seq received)
-
-* TAG_RECONNECT_RETRY_SESSION (server only): fail reconnect due to stale connect_seq
-
-* TAG_RECONNECT_RETRY_GLOBAL (server only): fail reconnect due to stale global_seq
-
-* TAG_RECONNECT_WAIT (server only): fail reconnect due to connect race.
-
- - Indicates that the server is already connecting to the client, and
- that direction should win the race. The client should wait for that
- connection to complete.
-
-Message exchange
-----------------
-
-Once a session is stablished, we can exchange messages.
-
-* TAG_MSG: a message::
-
- ceph_msg_header2
- front
- middle
- data
-
- - The ceph_msg_header is modified in ceph_msg_header2 to include an
- ack_seq. This avoids the need for a TAG_ACK message most of the time.
-
-* TAG_ACK: acknowledge receipt of message(s)::
-
- __le64 seq
-
- - This is only used for stateful sessions.
-
-* TAG_KEEPALIVE2: check for connection liveness::
-
- ceph_timespec stamp
-
- - Time stamp is local to sender.
-
-* TAG_KEEPALIVE2_ACK: reply to a keepalive2::
-
- ceph_timestamp stamp
-
- - Time stamp is from the TAG_KEEPALIVE2 we are responding to.
-
-* TAG_CLOSE: terminate a stream
-
- Indicates that a stream should be terminated. This is equivalent to
- a hangup or reset (i.e., should trigger ms_handle_reset). It isn't
- strictly necessary or useful if there is only a single stream as we
- could just disconnect the TCP connection, although one could
- certainly use it creatively (e.g., reset the stream state and retry
- an authentication handshake).
Code Review / stor4nfv.git / blobdiff