5 Ceph MON, OSD and MDS can generate heap profiles using
6 ``tcmalloc``. To generate heap profiles, ensure you have
7 ``google-perftools`` installed::
9 sudo apt-get install google-perftools
11 The profiler dumps output to your ``log file`` directory (i.e.,
12 ``/var/log/ceph``). See `Logging and Debugging`_ for details.
13 To view the profiler logs with Google's performance tools, execute the
16 google-pprof --text {path-to-daemon} {log-path/filename}
20 $ ceph tell osd.0 heap start_profiler
21 $ ceph tell osd.0 heap dump
22 osd.0 tcmalloc heap stats:------------------------------------------------
23 MALLOC: 2632288 ( 2.5 MiB) Bytes in use by application
24 MALLOC: + 499712 ( 0.5 MiB) Bytes in page heap freelist
25 MALLOC: + 543800 ( 0.5 MiB) Bytes in central cache freelist
26 MALLOC: + 327680 ( 0.3 MiB) Bytes in transfer cache freelist
27 MALLOC: + 1239400 ( 1.2 MiB) Bytes in thread cache freelists
28 MALLOC: + 1142936 ( 1.1 MiB) Bytes in malloc metadata
30 MALLOC: = 6385816 ( 6.1 MiB) Actual memory used (physical + swap)
31 MALLOC: + 0 ( 0.0 MiB) Bytes released to OS (aka unmapped)
33 MALLOC: = 6385816 ( 6.1 MiB) Virtual address space used
35 MALLOC: 231 Spans in use
36 MALLOC: 56 Thread heaps in use
37 MALLOC: 8192 Tcmalloc page size
38 ------------------------------------------------
39 Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
40 Bytes released to the OS take up virtual address space but no physical memory.
41 $ google-pprof --text \
43 /var/log/ceph/ceph-osd.0.profile.0001.heap
45 1.9 51.1% 51.1% 1.9 51.1% ceph::log::Log::create_entry
46 1.8 47.3% 98.4% 1.8 47.3% std::string::_Rep::_S_create
47 0.0 0.4% 98.9% 0.0 0.6% SimpleMessenger::add_accept_pipe
48 0.0 0.4% 99.2% 0.0 0.6% decode_message
51 Another heap dump on the same daemon will add another file. It is
52 convenient to compare to a previous heap dump to show what has grown
53 in the interval. For instance::
55 $ google-pprof --text --base out/osd.0.profile.0001.heap \
56 ceph-osd out/osd.0.profile.0003.heap
58 0.1 50.3% 50.3% 0.1 50.3% ceph::log::Log::create_entry
59 0.1 46.6% 96.8% 0.1 46.6% std::string::_Rep::_S_create
60 0.0 0.9% 97.7% 0.0 26.1% ReplicatedPG::do_op
61 0.0 0.8% 98.5% 0.0 0.8% __gnu_cxx::new_allocator::allocate
63 Refer to `Google Heap Profiler`_ for additional details.
65 Once you have the heap profiler installed, start your cluster and
66 begin using the heap profiler. You may enable or disable the heap
67 profiler at runtime, or ensure that it runs continuously. For the
68 following commandline usage, replace ``{daemon-type}`` with ``mon``,
69 ``osd`` or ``mds``, and replace ``{daemon-id}`` with the OSD number or
76 To start the heap profiler, execute the following::
78 ceph tell {daemon-type}.{daemon-id} heap start_profiler
82 ceph tell osd.1 heap start_profiler
84 Alternatively the profile can be started when the daemon starts
85 running if the ``CEPH_HEAP_PROFILER_INIT=true`` variable is found in
91 To print out statistics, execute the following::
93 ceph tell {daemon-type}.{daemon-id} heap stats
97 ceph tell osd.0 heap stats
99 .. note:: Printing stats does not require the profiler to be running and does
100 not dump the heap allocation information to a file.
103 Dumping Heap Information
104 ------------------------
106 To dump heap information, execute the following::
108 ceph tell {daemon-type}.{daemon-id} heap dump
112 ceph tell mds.a heap dump
114 .. note:: Dumping heap information only works when the profiler is running.
120 To release memory that ``tcmalloc`` has allocated but which is not being used by
121 the Ceph daemon itself, execute the following::
123 ceph tell {daemon-type}{daemon-id} heap release
127 ceph tell osd.2 heap release
130 Stopping the Profiler
131 ---------------------
133 To stop the heap profiler, execute the following::
135 ceph tell {daemon-type}.{daemon-id} heap stop_profiler
139 ceph tell osd.0 heap stop_profiler
141 .. _Logging and Debugging: ../log-and-debug
142 .. _Google Heap Profiler: http://goog-perftools.sourceforge.net/doc/heap_profiler.html