1 =======================
3 =======================
5 Typically, when you add debugging to your Ceph configuration, you do so at
6 runtime. You can also add Ceph debug logging to your Ceph configuration file if
7 you are encountering issues when starting your cluster. You may view Ceph log
8 files under ``/var/log/ceph`` (the default location).
10 .. tip:: When debug output slows down your system, the latency can hide
13 Logging is resource intensive. If you are encountering a problem in a specific
14 area of your cluster, enable logging for that area of the cluster. For example,
15 if your OSDs are running fine, but your metadata servers are not, you should
16 start by enabling debug logging for the specific metadata server instance(s)
17 giving you trouble. Enable logging for each subsystem as needed.
19 .. important:: Verbose logging can generate over 1GB of data per hour. If your
20 OS disk reaches its capacity, the node will stop working.
22 If you enable or increase the rate of Ceph logging, ensure that you have
23 sufficient disk space on your OS disk. See `Accelerating Log Rotation`_ for
24 details on rotating log files. When your system is running well, remove
25 unnecessary debugging settings to ensure your cluster runs optimally. Logging
26 debug output messages is relatively slow, and a waste of resources when
27 operating your cluster.
29 See `Subsystem, Log and Debug Settings`_ for details on available settings.
34 If you would like to see the configuration settings at runtime, you must log
35 in to a host with a running daemon and execute the following::
37 ceph daemon {daemon-name} config show | less
41 ceph daemon osd.0 config show | less
43 To activate Ceph's debugging output (*i.e.*, ``dout()``) at runtime, use the
44 ``ceph tell`` command to inject arguments into the runtime configuration::
46 ceph tell {daemon-type}.{daemon id or *} injectargs --{name} {value} [--{name} {value}]
48 Replace ``{daemon-type}`` with one of ``osd``, ``mon`` or ``mds``. You may apply
49 the runtime setting to all daemons of a particular type with ``*``, or specify
50 a specific daemon's ID. For example, to increase
51 debug logging for a ``ceph-osd`` daemon named ``osd.0``, execute the following::
53 ceph tell osd.0 injectargs --debug-osd 0/5
55 The ``ceph tell`` command goes through the monitors. If you cannot bind to the
56 monitor, you can still make the change by logging into the host of the daemon
57 whose configuration you'd like to change using ``ceph daemon``.
60 sudo ceph daemon osd.0 config set debug_osd 0/5
62 See `Subsystem, Log and Debug Settings`_ for details on available settings.
68 To activate Ceph's debugging output (*i.e.*, ``dout()``) at boot time, you must
69 add settings to your Ceph configuration file. Subsystems common to each daemon
70 may be set under ``[global]`` in your configuration file. Subsystems for
71 particular daemons are set under the daemon section in your configuration file
72 (*e.g.*, ``[mon]``, ``[osd]``, ``[mds]``). For example::
90 debug mds balancer = 1
93 See `Subsystem, Log and Debug Settings`_ for details.
96 Accelerating Log Rotation
97 =========================
99 If your OS disk is relatively full, you can accelerate log rotation by modifying
100 the Ceph log rotation file at ``/etc/logrotate.d/ceph``. Add a size setting
101 after the rotation frequency to accelerate log rotation (via cronjob) if your
102 logs exceed the size setting. For example, the default setting looks like
110 Modify it by adding a ``size`` setting. ::
118 Then, start the crontab editor for your user space. ::
122 Finally, add an entry to check the ``etc/logrotate.d/ceph`` file. ::
124 30 * * * * /usr/sbin/logrotate /etc/logrotate.d/ceph >/dev/null 2>&1
126 The preceding example checks the ``etc/logrotate.d/ceph`` file every 30 minutes.
132 Debugging may also require you to track down memory and threading issues.
133 You can run a single daemon, a type of daemon, or the whole cluster with
134 Valgrind. You should only use Valgrind when developing or debugging Ceph.
135 Valgrind is computationally expensive, and will slow down your system otherwise.
136 Valgrind messages are logged to ``stderr``.
139 Subsystem, Log and Debug Settings
140 =================================
142 In most cases, you will enable debug logging output via subsystems.
147 Each subsystem has a logging level for its output logs, and for its logs
148 in-memory. You may set different values for each of these subsystems by setting
149 a log file level and a memory level for debug logging. Ceph's logging levels
150 operate on a scale of ``1`` to ``20``, where ``1`` is terse and ``20`` is
151 verbose [#]_ . In general, the logs in-memory are not sent to the output log unless:
153 - a fatal signal is raised or
154 - an ``assert`` in source code is triggered or
155 - upon requested. Please consult `document on admin socket <http://docs.ceph.com/docs/master/man/8/ceph/#daemon>`_ for more details.
157 A debug logging setting can take a single value for the log level and the
158 memory level, which sets them both as the same value. For example, if you
159 specify ``debug ms = 5``, Ceph will treat it as a log level and a memory level
160 of ``5``. You may also specify them separately. The first setting is the log
161 level, and the second setting is the memory level. You must separate them with
162 a forward slash (/). For example, if you want to set the ``ms`` subsystem's
163 debug logging level to ``1`` and its memory level to ``5``, you would specify it
164 as ``debug ms = 1/5``. For example:
170 debug {subsystem} = {log-level}/{memory-level}
172 debug mds balancer = 1/20
175 The following table provides a list of Ceph subsystems and their default log and
176 memory levels. Once you complete your logging efforts, restore the subsystems
177 to their default level or to a level suitable for normal operations.
180 +--------------------+-----------+--------------+
181 | Subsystem | Log Level | Memory Level |
182 +====================+===========+==============+
183 | ``default`` | 0 | 5 |
184 +--------------------+-----------+--------------+
185 | ``lockdep`` | 0 | 5 |
186 +--------------------+-----------+--------------+
187 | ``context`` | 0 | 5 |
188 +--------------------+-----------+--------------+
189 | ``crush`` | 1 | 5 |
190 +--------------------+-----------+--------------+
192 +--------------------+-----------+--------------+
193 | ``mds balancer`` | 1 | 5 |
194 +--------------------+-----------+--------------+
195 | ``mds locker`` | 1 | 5 |
196 +--------------------+-----------+--------------+
197 | ``mds log`` | 1 | 5 |
198 +--------------------+-----------+--------------+
199 | ``mds log expire`` | 1 | 5 |
200 +--------------------+-----------+--------------+
201 | ``mds migrator`` | 1 | 5 |
202 +--------------------+-----------+--------------+
203 | ``buffer`` | 0 | 0 |
204 +--------------------+-----------+--------------+
205 | ``timer`` | 0 | 5 |
206 +--------------------+-----------+--------------+
207 | ``filer`` | 0 | 5 |
208 +--------------------+-----------+--------------+
209 | ``objecter`` | 0 | 0 |
210 +--------------------+-----------+--------------+
211 | ``rados`` | 0 | 5 |
212 +--------------------+-----------+--------------+
214 +--------------------+-----------+--------------+
215 | ``journaler`` | 0 | 5 |
216 +--------------------+-----------+--------------+
217 | ``objectcacher`` | 0 | 5 |
218 +--------------------+-----------+--------------+
219 | ``client`` | 0 | 5 |
220 +--------------------+-----------+--------------+
222 +--------------------+-----------+--------------+
223 | ``optracker`` | 0 | 5 |
224 +--------------------+-----------+--------------+
225 | ``objclass`` | 0 | 5 |
226 +--------------------+-----------+--------------+
227 | ``filestore`` | 1 | 5 |
228 +--------------------+-----------+--------------+
229 | ``journal`` | 1 | 5 |
230 +--------------------+-----------+--------------+
232 +--------------------+-----------+--------------+
234 +--------------------+-----------+--------------+
236 +--------------------+-----------+--------------+
237 | ``paxos`` | 0 | 5 |
238 +--------------------+-----------+--------------+
240 +--------------------+-----------+--------------+
242 +--------------------+-----------+--------------+
243 | ``finisher`` | 1 | 5 |
244 +--------------------+-----------+--------------+
245 | ``heartbeatmap`` | 1 | 5 |
246 +--------------------+-----------+--------------+
247 | ``perfcounter`` | 1 | 5 |
248 +--------------------+-----------+--------------+
250 +--------------------+-----------+--------------+
251 | ``javaclient`` | 1 | 5 |
252 +--------------------+-----------+--------------+
254 +--------------------+-----------+--------------+
255 | ``throttle`` | 1 | 5 |
256 +--------------------+-----------+--------------+
262 Logging and debugging settings are not required in a Ceph configuration file,
263 but you may override default settings as needed. Ceph supports the following
269 :Description: The location of the logging file for your cluster.
272 :Default: ``/var/log/ceph/$cluster-$name.log``
277 :Description: The maximum number of new log files.
285 :Description: The maximum number of recent events to include in a log file.
288 :Default: ``1000000``
293 :Description: Determines if logging messages should appear in ``stderr``.
301 :Description: Determines if error messages should appear in ``stderr``.
309 :Description: Determines if logging messages should appear in ``syslog``.
317 :Description: Determines if error messages should appear in ``syslog``.
323 ``log flush on exit``
325 :Description: Determines if Ceph should flush the log files after exit.
333 :Description: Determines if ``clog`` messages should be sent to monitors.
341 :Description: Determines if ``clog`` messages should be sent to syslog.
347 ``mon cluster log to syslog``
349 :Description: Determines if the cluster log should be output to the syslog.
355 ``mon cluster log file``
357 :Description: The location of the cluster's log file.
360 :Default: ``/var/log/ceph/$cluster.log``
368 ``osd debug drop ping probability``
376 ``osd debug drop ping duration``
383 ``osd debug drop pg create probability``
390 ``osd debug drop pg create duration``
398 ``osd tmapput sets uses tmap``
400 :Description: Uses ``tmap``. For debug only.
406 ``osd min pg log entries``
408 :Description: The minimum number of log entries for placement groups.
409 :Type: 32-bit Unsigned Integer
414 ``osd op log threshold``
416 :Description: How many op log messages to show up in one pass.
426 ``filestore debug omap check``
428 :Description: Debugging check on synchronization. This is an expensive operation.
438 ``mds debug scatterstat``
440 :Description: Ceph will assert that various recursive stat invariants are true
441 (for developers only).
450 :Description: Ceph will verify directory fragmentation invariants when
451 convenient (developers only).
458 ``mds debug auth pins``
460 :Description: The debug auth pin invariants (for developers only).
466 ``mds debug subtrees``
468 :Description: The debug subtree invariants (for developers only).
479 ``rgw log nonexistent bucket``
481 :Description: Should we log a non-existent buckets?
487 ``rgw log object name``
489 :Description: Should an object's name be logged. // man date to see codes (a subset are supported)
492 :Default: ``%Y-%m-%d-%H-%i-%n``
495 ``rgw log object name utc``
497 :Description: Object log name contains UTC?
503 ``rgw enable ops log``
505 :Description: Enables logging of every RGW operation.
511 ``rgw enable usage log``
513 :Description: Enable logging of RGW's bandwidth usage.
519 ``rgw usage log flush threshold``
521 :Description: Threshold to flush pending log data.
527 ``rgw usage log tick interval``
529 :Description: Flush pending log data every ``s`` seconds.
535 ``rgw intent log object name``
540 :Default: ``%Y-%m-%d-%i-%n``
543 ``rgw intent log object name utc``
545 :Description: Include a UTC timestamp in the intent log object name.
550 .. [#] there are levels >20 in some rare cases and that they are extremely verbose.