1 <?xml version="1.0" encoding="ISO-8859-1"?>
2 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
3 <html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"><head><!--
4 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
5 This file is generated from xml source: DO NOT EDIT
6 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
8 <title>Apache Performance Tuning - Apache HTTP Server</title>
9 <link href="../style/css/manual.css" rel="stylesheet" media="all" type="text/css" title="Main stylesheet" />
10 <link href="../style/css/manual-loose-100pc.css" rel="alternate stylesheet" media="all" type="text/css" title="No Sidebar - Default font size" />
11 <link href="../style/css/manual-print.css" rel="stylesheet" media="print" type="text/css" />
12 <link href="../images/favicon.ico" rel="shortcut icon" /></head>
13 <body id="manual-page"><div id="page-header">
14 <p class="menu"><a href="../mod/">Modules</a> | <a href="../mod/directives.html">Directives</a> | <a href="../faq/">FAQ</a> | <a href="../glossary.html">Glossary</a> | <a href="../sitemap.html">Sitemap</a></p>
15 <p class="apache">Apache HTTP Server Version 2.0</p>
16 <img alt="" src="../images/feather.gif" /></div>
17 <div class="up"><a href="./"><img title="<-" alt="<-" src="../images/left.gif" /></a></div>
19 <a href="http://www.apache.org/">Apache</a> > <a href="http://httpd.apache.org/">HTTP Server</a> > <a href="http://httpd.apache.org/docs/">Documentation</a> > <a href="../">Version 2.0</a> > <a href="./">Miscellaneous Documentation</a></div><div id="page-content"><div id="preamble"><h1>Apache Performance Tuning</h1>
21 <p><span>Available Languages: </span><a href="../en/misc/perf-tuning.html" title="English"> en </a> |
22 <a href="../ko/misc/perf-tuning.html" hreflang="ko" rel="alternate" title="Korean"> ko </a> |
23 <a href="../tr/misc/perf-tuning.html" hreflang="tr" rel="alternate" title="Türkçe"> tr </a></p>
27 <p>Apache 2.x is a general-purpose webserver, designed to
28 provide a balance of flexibility, portability, and performance.
29 Although it has not been designed specifically to set benchmark
30 records, Apache 2.x is capable of high performance in many
31 real-world situations.</p>
33 <p>Compared to Apache 1.3, release 2.x contains many additional
34 optimizations to increase throughput and scalability. Most of
35 these improvements are enabled by default. However, there are
36 compile-time and run-time configuration choices that can
37 significantly affect performance. This document describes the
38 options that a server administrator can configure to tune the
39 performance of an Apache 2.x installation. Some of these
40 configuration options enable the httpd to better take advantage
41 of the capabilities of the hardware and OS, while others allow
42 the administrator to trade functionality for speed.</p>
45 <div id="quickview"><ul id="toc"><li><img alt="" src="../images/down.gif" /> <a href="#hardware">Hardware and Operating System Issues</a></li>
46 <li><img alt="" src="../images/down.gif" /> <a href="#runtime">Run-Time Configuration Issues</a></li>
47 <li><img alt="" src="../images/down.gif" /> <a href="#compiletime">Compile-Time Configuration Issues</a></li>
48 <li><img alt="" src="../images/down.gif" /> <a href="#trace">Appendix: Detailed Analysis of a Trace</a></li>
50 <div class="top"><a href="#page-header"><img alt="top" src="../images/up.gif" /></a></div>
52 <h2><a name="hardware" id="hardware">Hardware and Operating System Issues</a></h2>
56 <p>The single biggest hardware issue affecting webserver
57 performance is RAM. A webserver should never ever have to swap,
58 as swapping increases the latency of each request beyond a point
59 that users consider "fast enough". This causes users to hit
60 stop and reload, further increasing the load. You can, and
61 should, control the <code class="directive"><a href="../mod/mpm_common.html#maxclients">MaxClients</a></code> setting so that your server
62 does not spawn so many children it starts swapping. This procedure
63 for doing this is simple: determine the size of your average Apache
64 process, by looking at your process list via a tool such as
65 <code>top</code>, and divide this into your total available memory,
66 leaving some room for other processes.</p>
68 <p>Beyond that the rest is mundane: get a fast enough CPU, a
69 fast enough network card, and fast enough disks, where "fast
70 enough" is something that needs to be determined by
73 <p>Operating system choice is largely a matter of local
74 concerns. But some guidelines that have proven generally
79 <p>Run the latest stable release and patchlevel of the
80 operating system that you choose. Many OS suppliers have
81 introduced significant performance improvements to their
82 TCP stacks and thread libraries in recent years.</p>
86 <p>If your OS supports a <code>sendfile(2)</code> system
87 call, make sure you install the release and/or patches
88 needed to enable it. (With Linux, for example, this means
89 using Linux 2.4 or later. For early releases of Solaris 8,
90 you may need to apply a patch.) On systems where it is
91 available, <code>sendfile</code> enables Apache 2 to deliver
92 static content faster and with lower CPU utilization.</p>
96 </div><div class="top"><a href="#page-header"><img alt="top" src="../images/up.gif" /></a></div>
98 <h2><a name="runtime" id="runtime">Run-Time Configuration Issues</a></h2>
102 <table class="related"><tr><th>Related Modules</th><th>Related Directives</th></tr><tr><td><ul><li><code class="module"><a href="../mod/mod_dir.html">mod_dir</a></code></li><li><code class="module"><a href="../mod/mpm_common.html">mpm_common</a></code></li><li><code class="module"><a href="../mod/mod_status.html">mod_status</a></code></li></ul></td><td><ul><li><code class="directive"><a href="../mod/core.html#allowoverride">AllowOverride</a></code></li><li><code class="directive"><a href="../mod/mod_dir.html#directoryindex">DirectoryIndex</a></code></li><li><code class="directive"><a href="../mod/core.html#hostnamelookups">HostnameLookups</a></code></li><li><code class="directive"><a href="../mod/core.html#enablemmap">EnableMMAP</a></code></li><li><code class="directive"><a href="../mod/core.html#enablesendfile">EnableSendfile</a></code></li><li><code class="directive"><a href="../mod/core.html#keepalivetimeout">KeepAliveTimeout</a></code></li><li><code class="directive"><a href="../mod/prefork.html#maxspareservers">MaxSpareServers</a></code></li><li><code class="directive"><a href="../mod/prefork.html#minspareservers">MinSpareServers</a></code></li><li><code class="directive"><a href="../mod/core.html#options">Options</a></code></li><li><code class="directive"><a href="../mod/mpm_common.html#startservers">StartServers</a></code></li></ul></td></tr></table>
104 <h3><a name="dns" id="dns">HostnameLookups and other DNS considerations</a></h3>
108 <p>Prior to Apache 1.3, <code class="directive"><a href="../mod/core.html#hostnamelookups">HostnameLookups</a></code> defaulted to <code>On</code>.
109 This adds latency to every request because it requires a
110 DNS lookup to complete before the request is finished. In
111 Apache 1.3 this setting defaults to <code>Off</code>. If you need
112 to have addresses in your log files resolved to hostnames, use the
113 <code class="program"><a href="../programs/logresolve.html">logresolve</a></code>
114 program that comes with Apache, or one of the numerous log
115 reporting packages which are available.</p>
117 <p>It is recommended that you do this sort of postprocessing of
118 your log files on some machine other than the production web
119 server machine, in order that this activity not adversely affect
120 server performance.</p>
122 <p>If you use any <code><code class="directive"><a href="../mod/mod_access.html#allow">Allow</a></code>
123 from domain</code> or <code><code class="directive"><a href="../mod/mod_access.html#deny">Deny</a></code> from domain</code>
124 directives (i.e., using a hostname, or a domain name, rather than
125 an IP address) then you will pay for
126 two DNS lookups (a reverse, followed by a forward lookup
127 to make sure that the reverse is not being spoofed). For best
128 performance, therefore, use IP addresses, rather than names, when
129 using these directives, if possible.</p>
131 <p>Note that it's possible to scope the directives, such as
132 within a <code><Location /server-status></code> section.
133 In this case the DNS lookups are only performed on requests
134 matching the criteria. Here's an example which disables lookups
135 except for <code>.html</code> and <code>.cgi</code> files:</p>
137 <div class="example"><p><code>
138 HostnameLookups off<br />
139 <Files ~ "\.(html|cgi)$"><br />
140 <span class="indent">
141 HostnameLookups on<br />
146 <p>But even still, if you just need DNS names in some CGIs you
147 could consider doing the <code>gethostbyname</code> call in the
148 specific CGIs that need it.</p>
152 <h3><a name="symlinks" id="symlinks">FollowSymLinks and SymLinksIfOwnerMatch</a></h3>
156 <p>Wherever in your URL-space you do not have an <code>Options
157 FollowSymLinks</code>, or you do have an <code>Options
158 SymLinksIfOwnerMatch</code> Apache will have to issue extra
159 system calls to check up on symlinks. One extra call per
160 filename component. For example, if you had:</p>
162 <div class="example"><p><code>
163 DocumentRoot /www/htdocs<br />
164 <Directory /><br />
165 <span class="indent">
166 Options SymLinksIfOwnerMatch<br />
171 <p>and a request is made for the URI <code>/index.html</code>.
172 Then Apache will perform <code>lstat(2)</code> on
173 <code>/www</code>, <code>/www/htdocs</code>, and
174 <code>/www/htdocs/index.html</code>. The results of these
175 <code>lstats</code> are never cached, so they will occur on
176 every single request. If you really desire the symlinks
177 security checking you can do something like this:</p>
179 <div class="example"><p><code>
180 DocumentRoot /www/htdocs<br />
181 <Directory /><br />
182 <span class="indent">
183 Options FollowSymLinks<br />
185 </Directory><br />
187 <Directory /www/htdocs><br />
188 <span class="indent">
189 Options -FollowSymLinks +SymLinksIfOwnerMatch<br />
194 <p>This at least avoids the extra checks for the
195 <code class="directive"><a href="../mod/core.html#documentroot">DocumentRoot</a></code> path.
196 Note that you'll need to add similar sections if you
197 have any <code class="directive"><a href="../mod/mod_alias.html#alias">Alias</a></code> or
198 <code class="directive"><a href="../mod/mod_rewrite.html#rewriterule">RewriteRule</a></code> paths
199 outside of your document root. For highest performance,
200 and no symlink protection, set <code>FollowSymLinks</code>
201 everywhere, and never set <code>SymLinksIfOwnerMatch</code>.</p>
205 <h3><a name="htacess" id="htacess">AllowOverride</a></h3>
209 <p>Wherever in your URL-space you allow overrides (typically
210 <code>.htaccess</code> files) Apache will attempt to open
211 <code>.htaccess</code> for each filename component. For
214 <div class="example"><p><code>
215 DocumentRoot /www/htdocs<br />
216 <Directory /><br />
217 <span class="indent">
218 AllowOverride all<br />
223 <p>and a request is made for the URI <code>/index.html</code>.
224 Then Apache will attempt to open <code>/.htaccess</code>,
225 <code>/www/.htaccess</code>, and
226 <code>/www/htdocs/.htaccess</code>. The solutions are similar
227 to the previous case of <code>Options FollowSymLinks</code>.
228 For highest performance use <code>AllowOverride None</code>
229 everywhere in your filesystem.</p>
233 <h3><a name="negotiation" id="negotiation">Negotiation</a></h3>
237 <p>If at all possible, avoid content-negotiation if you're
238 really interested in every last ounce of performance. In
239 practice the benefits of negotiation outweigh the performance
240 penalties. There's one case where you can speed up the server.
241 Instead of using a wildcard such as:</p>
243 <div class="example"><p><code>
247 <p>Use a complete list of options:</p>
249 <div class="example"><p><code>
250 DirectoryIndex index.cgi index.pl index.shtml index.html
253 <p>where you list the most common choice first.</p>
255 <p>Also note that explicitly creating a <code>type-map</code>
256 file provides better performance than using
257 <code>MultiViews</code>, as the necessary information can be
258 determined by reading this single file, rather than having to
259 scan the directory for files.</p>
261 <p>If your site needs content negotiation consider using
262 <code>type-map</code> files, rather than the <code>Options
263 MultiViews</code> directive to accomplish the negotiation. See the
264 <a href="../content-negotiation.html">Content Negotiation</a>
265 documentation for a full discussion of the methods of negotiation,
266 and instructions for creating <code>type-map</code> files.</p>
270 <h3>Memory-mapping</h3>
274 <p>In situations where Apache 2.x needs to look at the contents
275 of a file being delivered--for example, when doing server-side-include
276 processing--it normally memory-maps the file if the OS supports
277 some form of <code>mmap(2)</code>.</p>
279 <p>On some platforms, this memory-mapping improves performance.
280 However, there are cases where memory-mapping can hurt the performance
281 or even the stability of the httpd:</p>
285 <p>On some operating systems, <code>mmap</code> does not scale
286 as well as <code>read(2)</code> when the number of CPUs increases.
287 On multiprocessor Solaris servers, for example, Apache 2.x sometimes
288 delivers server-parsed files faster when <code>mmap</code> is disabled.</p>
292 <p>If you memory-map a file located on an NFS-mounted filesystem
293 and a process on another NFS client machine deletes or truncates
294 the file, your process may get a bus error the next time it tries
295 to access the mapped file content.</p>
299 <p>For installations where either of these factors applies, you
300 should use <code>EnableMMAP off</code> to disable the memory-mapping
301 of delivered files. (Note: This directive can be overridden on
302 a per-directory basis.)</p>
310 <p>In situations where Apache 2.x can ignore the contents of the file
311 to be delivered -- for example, when serving static file content --
312 it normally uses the kernel sendfile support the file if the OS
313 supports the <code>sendfile(2)</code> operation.</p>
315 <p>On most platforms, using sendfile improves performance by eliminating
316 separate read and send mechanics. However, there are cases where using
317 sendfile can harm the stability of the httpd:</p>
321 <p>Some platforms may have broken sendfile support that the build
322 system did not detect, especially if the binaries were built on
323 another box and moved to such a machine with broken sendfile support.</p>
326 <p>With an NFS-mounted files, the kernel may be unable
327 to reliably serve the network file through it's own cache.</p>
331 <p>For installations where either of these factors applies, you
332 should use <code>EnableSendfile off</code> to disable sendfile
333 delivery of file contents. (Note: This directive can be overridden
334 on a per-directory basis.)</p>
338 <h3><a name="process" id="process">Process Creation</a></h3>
342 <p>Prior to Apache 1.3 the <code class="directive"><a href="../mod/prefork.html#minspareservers">MinSpareServers</a></code>, <code class="directive"><a href="../mod/prefork.html#maxspareservers">MaxSpareServers</a></code>, and <code class="directive"><a href="../mod/mpm_common.html#startservers">StartServers</a></code> settings all had drastic effects on
343 benchmark results. In particular, Apache required a "ramp-up"
344 period in order to reach a number of children sufficient to serve
345 the load being applied. After the initial spawning of
346 <code class="directive"><a href="../mod/mpm_common.html#startservers">StartServers</a></code> children,
347 only one child per second would be created to satisfy the
348 <code class="directive"><a href="../mod/prefork.html#minspareservers">MinSpareServers</a></code>
349 setting. So a server being accessed by 100 simultaneous
350 clients, using the default <code class="directive"><a href="../mod/mpm_common.html#startservers">StartServers</a></code> of <code>5</code> would take on
351 the order 95 seconds to spawn enough children to handle
352 the load. This works fine in practice on real-life servers,
353 because they aren't restarted frequently. But does really
354 poorly on benchmarks which might only run for ten minutes.</p>
356 <p>The one-per-second rule was implemented in an effort to
357 avoid swamping the machine with the startup of new children. If
358 the machine is busy spawning children it can't service
359 requests. But it has such a drastic effect on the perceived
360 performance of Apache that it had to be replaced. As of Apache
361 1.3, the code will relax the one-per-second rule. It will spawn
362 one, wait a second, then spawn two, wait a second, then spawn
363 four, and it will continue exponentially until it is spawning
364 32 children per second. It will stop whenever it satisfies the
365 <code class="directive"><a href="../mod/prefork.html#minspareservers">MinSpareServers</a></code>
368 <p>This appears to be responsive enough that it's almost
369 unnecessary to twiddle the <code class="directive"><a href="../mod/prefork.html#minspareservers">MinSpareServers</a></code>, <code class="directive"><a href="../mod/prefork.html#maxspareservers">MaxSpareServers</a></code> and <code class="directive"><a href="../mod/mpm_common.html#startservers">StartServers</a></code> knobs. When more than 4 children are
370 spawned per second, a message will be emitted to the
371 <code class="directive"><a href="../mod/core.html#errorlog">ErrorLog</a></code>. If you
372 see a lot of these errors then consider tuning these settings.
373 Use the <code class="module"><a href="../mod/mod_status.html">mod_status</a></code> output as a guide.</p>
375 <p>Related to process creation is process death induced by the
376 <code class="directive"><a href="../mod/mpm_common.html#maxrequestsperchild">MaxRequestsPerChild</a></code>
377 setting. By default this is <code>0</code>,
378 which means that there is no limit to the number of requests
379 handled per child. If your configuration currently has this set
380 to some very low number, such as <code>30</code>, you may want to bump this
381 up significantly. If you are running SunOS or an old version of
382 Solaris, limit this to <code>10000</code> or so because of memory leaks.</p>
384 <p>When keep-alives are in use, children will be kept busy
385 doing nothing waiting for more requests on the already open
386 connection. The default <code class="directive"><a href="../mod/core.html#keepalivetimeout">KeepAliveTimeout</a></code> of <code>15</code>
387 seconds attempts to minimize this effect. The tradeoff here is
388 between network bandwidth and server resources. In no event
389 should you raise this above about <code>60</code> seconds, as <a href="http://www.research.digital.com/wrl/techreports/abstracts/95.4.html">
390 most of the benefits are lost</a>.</p>
394 </div><div class="top"><a href="#page-header"><img alt="top" src="../images/up.gif" /></a></div>
395 <div class="section">
396 <h2><a name="compiletime" id="compiletime">Compile-Time Configuration Issues</a></h2>
400 <h3>Choosing an MPM</h3>
404 <p>Apache 2.x supports pluggable concurrency models, called
405 <a href="../mpm.html">Multi-Processing Modules</a> (MPMs).
406 When building Apache, you must choose an MPM to use. There
407 are platform-specific MPMs for some platforms:
408 <code class="module"><a href="../mod/beos.html">beos</a></code>, <code class="module"><a href="../mod/mpm_netware.html">mpm_netware</a></code>,
409 <code class="module"><a href="../mod/mpmt_os2.html">mpmt_os2</a></code>, and <code class="module"><a href="../mod/mpm_winnt.html">mpm_winnt</a></code>. For
410 general Unix-type systems, there are several MPMs from which
411 to choose. The choice of MPM can affect the speed and scalability
416 <li>The <code class="module"><a href="../mod/worker.html">worker</a></code> MPM uses multiple child
417 processes with many threads each. Each thread handles
418 one connection at a time. Worker generally is a good
419 choice for high-traffic servers because it has a smaller
420 memory footprint than the prefork MPM.</li>
422 <li>The <code class="module"><a href="../mod/prefork.html">prefork</a></code> MPM uses multiple child
423 processes with one thread each. Each process handles
424 one connection at a time. On many systems, prefork is
425 comparable in speed to worker, but it uses more memory.
426 Prefork's threadless design has advantages over worker
427 in some situations: it can be used with non-thread-safe
428 third-party modules, and it is easier to debug on platforms
429 with poor thread debugging support.</li>
433 <p>For more information on these and other MPMs, please
434 see the MPM <a href="../mpm.html">documentation</a>.</p>
438 <h3><a name="modules" id="modules">Modules</a></h3>
442 <p>Since memory usage is such an important consideration in
443 performance, you should attempt to eliminate modules that you are
444 not actually using. If you have built the modules as <a href="../dso.html">DSOs</a>, eliminating modules is a simple
445 matter of commenting out the associated <code class="directive"><a href="../mod/mod_so.html#loadmodule">LoadModule</a></code> directive for that module.
446 This allows you to experiment with removing modules, and seeing
447 if your site still functions in their absense.</p>
449 <p>If, on the other hand, you have modules statically linked
450 into your Apache binary, you will need to recompile Apache in
451 order to remove unwanted modules.</p>
453 <p>An associated question that arises here is, of course, what
454 modules you need, and which ones you don't. The answer here
455 will, of course, vary from one web site to another. However, the
456 <em>minimal</em> list of modules which you can get by with tends
457 to include <code class="module"><a href="../mod/mod_mime.html">mod_mime</a></code>, <code class="module"><a href="../mod/mod_dir.html">mod_dir</a></code>,
458 and <code class="module"><a href="../mod/mod_log_config.html">mod_log_config</a></code>. <code>mod_log_config</code> is,
459 of course, optional, as you can run a web site without log
460 files. This is, however, not recommended.</p>
464 <h3>Atomic Operations</h3>
468 <p>Some modules, such as <code class="module"><a href="../mod/mod_cache.html">mod_cache</a></code> and
469 recent development builds of the worker MPM, use APR's
470 atomic API. This API provides atomic operations that can
471 be used for lightweight thread synchronization.</p>
473 <p>By default, APR implements these operations using the
474 most efficient mechanism available on each target
475 OS/CPU platform. Many modern CPUs, for example, have
476 an instruction that does an atomic compare-and-swap (CAS)
477 operation in hardware. On some platforms, however, APR
478 defaults to a slower, mutex-based implementation of the
479 atomic API in order to ensure compatibility with older
480 CPU models that lack such instructions. If you are
481 building Apache for one of these platforms, and you plan
482 to run only on newer CPUs, you can select a faster atomic
483 implementation at build time by configuring Apache with
484 the <code>--enable-nonportable-atomics</code> option:</p>
486 <div class="example"><p><code>
488 ./configure --with-mpm=worker --enable-nonportable-atomics=yes
491 <p>The <code>--enable-nonportable-atomics</code> option is
492 relevant for the following platforms:</p>
496 <li>Solaris on SPARC<br />
497 By default, APR uses mutex-based atomics on Solaris/SPARC.
498 If you configure with <code>--enable-nonportable-atomics</code>,
499 however, APR generates code that uses a SPARC v8plus opcode for
500 fast hardware compare-and-swap. If you configure Apache with
501 this option, the atomic operations will be more efficient
502 (allowing for lower CPU utilization and higher concurrency),
503 but the resulting executable will run only on UltraSPARC
507 <li>Linux on x86<br />
508 By default, APR uses mutex-based atomics on Linux. If you
509 configure with <code>--enable-nonportable-atomics</code>,
510 however, APR generates code that uses a 486 opcode for fast
511 hardware compare-and-swap. This will result in more efficient
512 atomic operations, but the resulting executable will run only
513 on 486 and later chips (and not on 386).
520 <h3>mod_status and ExtendedStatus On</h3>
524 <p>If you include <code class="module"><a href="../mod/mod_status.html">mod_status</a></code> and you also set
525 <code>ExtendedStatus On</code> when building and running
526 Apache, then on every request Apache will perform two calls to
527 <code>gettimeofday(2)</code> (or <code>times(2)</code>
528 depending on your operating system), and (pre-1.3) several
529 extra calls to <code>time(2)</code>. This is all done so that
530 the status report contains timing indications. For highest
531 performance, set <code>ExtendedStatus off</code> (which is the
536 <h3>accept Serialization - multiple sockets</h3>
540 <div class="warning"><h3>Warning:</h3>
541 <p>This section has not been fully updated
542 to take into account changes made in the 2.x version of the
543 Apache HTTP Server. Some of the information may still be
544 relevant, but please use it with care.</p>
547 <p>This discusses a shortcoming in the Unix socket API. Suppose
548 your web server uses multiple <code class="directive"><a href="../mod/mpm_common.html#listen">Listen</a></code> statements to listen on either multiple
549 ports or multiple addresses. In order to test each socket
550 to see if a connection is ready Apache uses
551 <code>select(2)</code>. <code>select(2)</code> indicates that a
552 socket has <em>zero</em> or <em>at least one</em> connection
553 waiting on it. Apache's model includes multiple children, and
554 all the idle ones test for new connections at the same time. A
555 naive implementation looks something like this (these examples
556 do not match the code, they're contrived for pedagogical
559 <div class="example"><p><code>
561 <span class="indent">
563 <span class="indent">
564 fd_set accept_fds;<br />
566 FD_ZERO (&accept_fds);<br />
567 for (i = first_socket; i <= last_socket; ++i) {<br />
568 <span class="indent">
569 FD_SET (i, &accept_fds);<br />
572 rc = select (last_socket+1, &accept_fds, NULL, NULL, NULL);<br />
573 if (rc < 1) continue;<br />
574 new_connection = -1;<br />
575 for (i = first_socket; i <= last_socket; ++i) {<br />
576 <span class="indent">
577 if (FD_ISSET (i, &accept_fds)) {<br />
578 <span class="indent">
579 new_connection = accept (i, NULL, NULL);<br />
580 if (new_connection != -1) break;<br />
585 if (new_connection != -1) break;<br />
588 process the new_connection;<br />
593 <p>But this naive implementation has a serious starvation problem.
594 Recall that multiple children execute this loop at the same
595 time, and so multiple children will block at
596 <code>select</code> when they are in between requests. All
597 those blocked children will awaken and return from
598 <code>select</code> when a single request appears on any socket
599 (the number of children which awaken varies depending on the
600 operating system and timing issues). They will all then fall
601 down into the loop and try to <code>accept</code> the
602 connection. But only one will succeed (assuming there's still
603 only one connection ready), the rest will be <em>blocked</em>
604 in <code>accept</code>. This effectively locks those children
605 into serving requests from that one socket and no other
606 sockets, and they'll be stuck there until enough new requests
607 appear on that socket to wake them all up. This starvation
608 problem was first documented in <a href="http://bugs.apache.org/index/full/467">PR#467</a>. There
609 are at least two solutions.</p>
611 <p>One solution is to make the sockets non-blocking. In this
612 case the <code>accept</code> won't block the children, and they
613 will be allowed to continue immediately. But this wastes CPU
614 time. Suppose you have ten idle children in
615 <code>select</code>, and one connection arrives. Then nine of
616 those children will wake up, try to <code>accept</code> the
617 connection, fail, and loop back into <code>select</code>,
618 accomplishing nothing. Meanwhile none of those children are
619 servicing requests that occurred on other sockets until they
620 get back up to the <code>select</code> again. Overall this
621 solution does not seem very fruitful unless you have as many
622 idle CPUs (in a multiprocessor box) as you have idle children,
623 not a very likely situation.</p>
625 <p>Another solution, the one used by Apache, is to serialize
626 entry into the inner loop. The loop looks like this
627 (differences highlighted):</p>
629 <div class="example"><p><code>
631 <span class="indent">
632 <strong>accept_mutex_on ();</strong><br />
634 <span class="indent">
635 fd_set accept_fds;<br />
637 FD_ZERO (&accept_fds);<br />
638 for (i = first_socket; i <= last_socket; ++i) {<br />
639 <span class="indent">
640 FD_SET (i, &accept_fds);<br />
643 rc = select (last_socket+1, &accept_fds, NULL, NULL, NULL);<br />
644 if (rc < 1) continue;<br />
645 new_connection = -1;<br />
646 for (i = first_socket; i <= last_socket; ++i) {<br />
647 <span class="indent">
648 if (FD_ISSET (i, &accept_fds)) {<br />
649 <span class="indent">
650 new_connection = accept (i, NULL, NULL);<br />
651 if (new_connection != -1) break;<br />
656 if (new_connection != -1) break;<br />
659 <strong>accept_mutex_off ();</strong><br />
660 process the new_connection;<br />
665 <p><a id="serialize" name="serialize">The functions</a>
666 <code>accept_mutex_on</code> and <code>accept_mutex_off</code>
667 implement a mutual exclusion semaphore. Only one child can have
668 the mutex at any time. There are several choices for
669 implementing these mutexes. The choice is defined in
670 <code>src/conf.h</code> (pre-1.3) or
671 <code>src/include/ap_config.h</code> (1.3 or later). Some
672 architectures do not have any locking choice made, on these
673 architectures it is unsafe to use multiple
674 <code class="directive"><a href="../mod/mpm_common.html#listen">Listen</a></code>
677 <p>The directive <code class="directive"><a href="../mod/mpm_common.html#acceptmutex">AcceptMutex</a></code> can be used to
678 change the selected mutex implementation at run-time.</p>
681 <dt><code>AcceptMutex flock</code></dt>
684 <p>This method uses the <code>flock(2)</code> system call to
685 lock a lock file (located by the <code class="directive"><a href="../mod/mpm_common.html#lockfile">LockFile</a></code> directive).</p>
688 <dt><code>AcceptMutex fcntl</code></dt>
691 <p>This method uses the <code>fcntl(2)</code> system call to
692 lock a lock file (located by the <code class="directive"><a href="../mod/mpm_common.html#lockfile">LockFile</a></code> directive).</p>
695 <dt><code>AcceptMutex sysvsem</code></dt>
698 <p>(1.3 or later) This method uses SysV-style semaphores to
699 implement the mutex. Unfortunately SysV-style semaphores have
700 some bad side-effects. One is that it's possible Apache will
701 die without cleaning up the semaphore (see the
702 <code>ipcs(8)</code> man page). The other is that the
703 semaphore API allows for a denial of service attack by any
704 CGIs running under the same uid as the webserver
705 (<em>i.e.</em>, all CGIs, unless you use something like
706 <code class="program"><a href="../programs/suexec.html">suexec</a></code> or <code>cgiwrapper</code>). For these
707 reasons this method is not used on any architecture except
708 IRIX (where the previous two are prohibitively expensive
709 on most IRIX boxes).</p>
712 <dt><code>AcceptMutex pthread</code></dt>
715 <p>(1.3 or later) This method uses POSIX mutexes and should
716 work on any architecture implementing the full POSIX threads
717 specification, however appears to only work on Solaris (2.5
718 or later), and even then only in certain configurations. If
719 you experiment with this you should watch out for your server
720 hanging and not responding. Static content only servers may
724 <dt><code>AcceptMutex posixsem</code></dt>
727 <p>(2.0 or later) This method uses POSIX semaphores. The
728 semaphore ownership is not recovered if a thread in the process
729 holding the mutex segfaults, resulting in a hang of the web
735 <p>If your system has another method of serialization which
736 isn't in the above list then it may be worthwhile adding code
739 <p>Another solution that has been considered but never
740 implemented is to partially serialize the loop -- that is, let
741 in a certain number of processes. This would only be of
742 interest on multiprocessor boxes where it's possible multiple
743 children could run simultaneously, and the serialization
744 actually doesn't take advantage of the full bandwidth. This is
745 a possible area of future investigation, but priority remains
746 low because highly parallel web servers are not the norm.</p>
748 <p>Ideally you should run servers without multiple
749 <code class="directive"><a href="../mod/mpm_common.html#listen">Listen</a></code>
750 statements if you want the highest performance.
755 <h3>accept Serialization - single socket</h3>
759 <p>The above is fine and dandy for multiple socket servers, but
760 what about single socket servers? In theory they shouldn't
761 experience any of these same problems because all children can
762 just block in <code>accept(2)</code> until a connection
763 arrives, and no starvation results. In practice this hides
764 almost the same "spinning" behaviour discussed above in the
765 non-blocking solution. The way that most TCP stacks are
766 implemented, the kernel actually wakes up all processes blocked
767 in <code>accept</code> when a single connection arrives. One of
768 those processes gets the connection and returns to user-space,
769 the rest spin in the kernel and go back to sleep when they
770 discover there's no connection for them. This spinning is
771 hidden from the user-land code, but it's there nonetheless.
772 This can result in the same load-spiking wasteful behaviour
773 that a non-blocking solution to the multiple sockets case
776 <p>For this reason we have found that many architectures behave
777 more "nicely" if we serialize even the single socket case. So
778 this is actually the default in almost all cases. Crude
779 experiments under Linux (2.0.30 on a dual Pentium pro 166
780 w/128Mb RAM) have shown that the serialization of the single
781 socket case causes less than a 3% decrease in requests per
782 second over unserialized single-socket. But unserialized
783 single-socket showed an extra 100ms latency on each request.
784 This latency is probably a wash on long haul lines, and only an
785 issue on LANs. If you want to override the single socket
786 serialization you can define
787 <code>SINGLE_LISTEN_UNSERIALIZED_ACCEPT</code> and then
788 single-socket servers will not serialize at all.</p>
792 <h3>Lingering Close</h3>
796 <p>As discussed in <a href="http://www.ics.uci.edu/pub/ietf/http/draft-ietf-http-connection-00.txt">
797 draft-ietf-http-connection-00.txt</a> section 8, in order for
798 an HTTP server to <strong>reliably</strong> implement the
799 protocol it needs to shutdown each direction of the
800 communication independently (recall that a TCP connection is
801 bi-directional, each half is independent of the other). This
802 fact is often overlooked by other servers, but is correctly
803 implemented in Apache as of 1.2.</p>
805 <p>When this feature was added to Apache it caused a flurry of
806 problems on various versions of Unix because of a
807 shortsightedness. The TCP specification does not state that the
808 <code>FIN_WAIT_2</code> state has a timeout, but it doesn't prohibit it.
809 On systems without the timeout, Apache 1.2 induces many sockets
810 stuck forever in the <code>FIN_WAIT_2</code> state. In many cases this
811 can be avoided by simply upgrading to the latest TCP/IP patches
812 supplied by the vendor. In cases where the vendor has never
813 released patches (<em>i.e.</em>, SunOS4 -- although folks with
814 a source license can patch it themselves) we have decided to
815 disable this feature.</p>
817 <p>There are two ways of accomplishing this. One is the socket
818 option <code>SO_LINGER</code>. But as fate would have it, this
819 has never been implemented properly in most TCP/IP stacks. Even
820 on those stacks with a proper implementation (<em>i.e.</em>,
821 Linux 2.0.31) this method proves to be more expensive (cputime)
822 than the next solution.</p>
824 <p>For the most part, Apache implements this in a function
825 called <code>lingering_close</code> (in
826 <code>http_main.c</code>). The function looks roughly like
829 <div class="example"><p><code>
830 void lingering_close (int s)<br />
832 <span class="indent">
833 char junk_buffer[2048];<br />
835 /* shutdown the sending side */<br />
836 shutdown (s, 1);<br />
838 signal (SIGALRM, lingering_death);<br />
842 <span class="indent">
843 select (s for reading, 2 second timeout);<br />
844 if (error) break;<br />
845 if (s is ready for reading) {<br />
846 <span class="indent">
847 if (read (s, junk_buffer, sizeof (junk_buffer)) <= 0) {<br />
848 <span class="indent">
852 /* just toss away whatever is here */<br />
863 <p>This naturally adds some expense at the end of a connection,
864 but it is required for a reliable implementation. As HTTP/1.1
865 becomes more prevalent, and all connections are persistent,
866 this expense will be amortized over more requests. If you want
867 to play with fire and disable this feature you can define
868 <code>NO_LINGCLOSE</code>, but this is not recommended at all.
869 In particular, as HTTP/1.1 pipelined persistent connections
870 come into use <code>lingering_close</code> is an absolute
871 necessity (and <a href="http://www.w3.org/Protocols/HTTP/Performance/Pipeline.html">
872 pipelined connections are faster</a>, so you want to support
877 <h3>Scoreboard File</h3>
881 <p>Apache's parent and children communicate with each other
882 through something called the scoreboard. Ideally this should be
883 implemented in shared memory. For those operating systems that
884 we either have access to, or have been given detailed ports
885 for, it typically is implemented using shared memory. The rest
886 default to using an on-disk file. The on-disk file is not only
887 slow, but it is unreliable (and less featured). Peruse the
888 <code>src/main/conf.h</code> file for your architecture and
889 look for either <code>USE_MMAP_SCOREBOARD</code> or
890 <code>USE_SHMGET_SCOREBOARD</code>. Defining one of those two
891 (as well as their companions <code>HAVE_MMAP</code> and
892 <code>HAVE_SHMGET</code> respectively) enables the supplied
893 shared memory code. If your system has another type of shared
894 memory, edit the file <code>src/main/http_main.c</code> and add
895 the hooks necessary to use it in Apache. (Send us back a patch
898 <div class="note">Historical note: The Linux port of Apache didn't start to
899 use shared memory until version 1.2 of Apache. This oversight
900 resulted in really poor and unreliable behaviour of earlier
901 versions of Apache on Linux.</div>
905 <h3>DYNAMIC_MODULE_LIMIT</h3>
909 <p>If you have no intention of using dynamically loaded modules
910 (you probably don't if you're reading this and tuning your
911 server for every last ounce of performance) then you should add
912 <code>-DDYNAMIC_MODULE_LIMIT=0</code> when building your
913 server. This will save RAM that's allocated only for supporting
914 dynamically loaded modules.</p>
918 </div><div class="top"><a href="#page-header"><img alt="top" src="../images/up.gif" /></a></div>
919 <div class="section">
920 <h2><a name="trace" id="trace">Appendix: Detailed Analysis of a Trace</a></h2>
924 <p>Here is a system call trace of Apache 2.0.38 with the worker MPM
925 on Solaris 8. This trace was collected using:</p>
927 <div class="example"><p><code>
928 truss -l -p <var>httpd_child_pid</var>.
931 <p>The <code>-l</code> option tells truss to log the ID of the
932 LWP (lightweight process--Solaris's form of kernel-level thread)
933 that invokes each system call.</p>
935 <p>Other systems may have different system call tracing utilities
936 such as <code>strace</code>, <code>ktrace</code>, or <code>par</code>.
937 They all produce similar output.</p>
939 <p>In this trace, a client has requested a 10KB static file
940 from the httpd. Traces of non-static requests or requests
941 with content negotiation look wildly different (and quite ugly
944 <div class="example"><pre>/67: accept(3, 0x00200BEC, 0x00200C0C, 1) (sleeping...)
945 /67: accept(3, 0x00200BEC, 0x00200C0C, 1) = 9</pre></div>
947 <p>In this trace, the listener thread is running within LWP #67.</p>
949 <div class="note">Note the lack of <code>accept(2)</code> serialization. On this
950 particular platform, the worker MPM uses an unserialized accept by
951 default unless it is listening on multiple ports.</div>
953 <div class="example"><pre>/65: lwp_park(0x00000000, 0) = 0
954 /67: lwp_unpark(65, 1) = 0</pre></div>
956 <p>Upon accepting the connection, the listener thread wakes up
957 a worker thread to do the request processing. In this trace,
958 the worker thread that handles the request is mapped to LWP #65.</p>
960 <div class="example"><pre>/65: getsockname(9, 0x00200BA4, 0x00200BC4, 1) = 0</pre></div>
962 <p>In order to implement virtual hosts, Apache needs to know
963 the local socket address used to accept the connection. It
964 is possible to eliminate this call in many situations (such
965 as when there are no virtual hosts, or when
966 <code class="directive"><a href="../mod/mpm_common.html#listen">Listen</a></code> directives
967 are used which do not have wildcard addresses). But
968 no effort has yet been made to do these optimizations. </p>
970 <div class="example"><pre>/65: brk(0x002170E8) = 0
971 /65: brk(0x002190E8) = 0</pre></div>
973 <p>The <code>brk(2)</code> calls allocate memory from the heap.
974 It is rare to see these in a system call trace, because the httpd
975 uses custom memory allocators (<code>apr_pool</code> and
976 <code>apr_bucket_alloc</code>) for most request processing.
977 In this trace, the httpd has just been started, so it must
978 call <code>malloc(3)</code> to get the blocks of raw memory
979 with which to create the custom memory allocators.</p>
981 <div class="example"><pre>/65: fcntl(9, F_GETFL, 0x00000000) = 2
982 /65: fstat64(9, 0xFAF7B818) = 0
983 /65: getsockopt(9, 65535, 8192, 0xFAF7B918, 0xFAF7B910, 2190656) = 0
984 /65: fstat64(9, 0xFAF7B818) = 0
985 /65: getsockopt(9, 65535, 8192, 0xFAF7B918, 0xFAF7B914, 2190656) = 0
986 /65: setsockopt(9, 65535, 8192, 0xFAF7B918, 4, 2190656) = 0
987 /65: fcntl(9, F_SETFL, 0x00000082) = 0</pre></div>
989 <p>Next, the worker thread puts the connection to the client (file
990 descriptor 9) in non-blocking mode. The <code>setsockopt(2)</code>
991 and <code>getsockopt(2)</code> calls are a side-effect of how
992 Solaris's libc handles <code>fcntl(2)</code> on sockets.</p>
994 <div class="example"><pre>/65: read(9, " G E T / 1 0 k . h t m".., 8000) = 97</pre></div>
996 <p>The worker thread reads the request from the client.</p>
998 <div class="example"><pre>/65: stat("/var/httpd/apache/httpd-8999/htdocs/10k.html", 0xFAF7B978) = 0
999 /65: open("/var/httpd/apache/httpd-8999/htdocs/10k.html", O_RDONLY) = 10</pre></div>
1001 <p>This httpd has been configured with <code>Options FollowSymLinks</code>
1002 and <code>AllowOverride None</code>. Thus it doesn't need to
1003 <code>lstat(2)</code> each directory in the path leading up to the
1004 requested file, nor check for <code>.htaccess</code> files.
1005 It simply calls <code>stat(2)</code> to verify that the file:
1006 1) exists, and 2) is a regular file, not a directory.</p>
1008 <div class="example"><pre>/65: sendfilev(0, 9, 0x00200F90, 2, 0xFAF7B53C) = 10269</pre></div>
1010 <p>In this example, the httpd is able to send the HTTP response
1011 header and the requested file with a single <code>sendfilev(2)</code>
1012 system call. Sendfile semantics vary among operating systems. On some other
1013 systems, it is necessary to do a <code>write(2)</code> or
1014 <code>writev(2)</code> call to send the headers before calling
1015 <code>sendfile(2)</code>.</p>
1017 <div class="example"><pre>/65: write(4, " 1 2 7 . 0 . 0 . 1 - ".., 78) = 78</pre></div>
1019 <p>This <code>write(2)</code> call records the request in the
1020 access log. Note that one thing missing from this trace is a
1021 <code>time(2)</code> call. Unlike Apache 1.3, Apache 2.x uses
1022 <code>gettimeofday(3)</code> to look up the time. On some operating
1023 systems, like Linux or Solaris, <code>gettimeofday</code> has an
1024 optimized implementation that doesn't require as much overhead
1025 as a typical system call.</p>
1027 <div class="example"><pre>/65: shutdown(9, 1, 1) = 0
1028 /65: poll(0xFAF7B980, 1, 2000) = 1
1029 /65: read(9, 0xFAF7BC20, 512) = 0
1030 /65: close(9) = 0</pre></div>
1032 <p>The worker thread does a lingering close of the connection.</p>
1034 <div class="example"><pre>/65: close(10) = 0
1035 /65: lwp_park(0x00000000, 0) (sleeping...)</pre></div>
1037 <p>Finally the worker thread closes the file that it has just delivered
1038 and blocks until the listener assigns it another connection.</p>
1040 <div class="example"><pre>/67: accept(3, 0x001FEB74, 0x001FEB94, 1) (sleeping...)</pre></div>
1042 <p>Meanwhile, the listener thread is able to accept another connection
1043 as soon as it has dispatched this connection to a worker thread (subject
1044 to some flow-control logic in the worker MPM that throttles the listener
1045 if all the available workers are busy). Though it isn't apparent from
1046 this trace, the next <code>accept(2)</code> can (and usually does, under
1047 high load conditions) occur in parallel with the worker thread's handling
1048 of the just-accepted connection.</p>
1051 <div class="bottomlang">
1052 <p><span>Available Languages: </span><a href="../en/misc/perf-tuning.html" title="English"> en </a> |
1053 <a href="../ko/misc/perf-tuning.html" hreflang="ko" rel="alternate" title="Korean"> ko </a> |
1054 <a href="../tr/misc/perf-tuning.html" hreflang="tr" rel="alternate" title="Türkçe"> tr </a></p>
1055 </div><div id="footer">
1056 <p class="apache">Copyright 2009 The Apache Software Foundation.<br />Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.</p>
1057 <p class="menu"><a href="../mod/">Modules</a> | <a href="../mod/directives.html">Directives</a> | <a href="../faq/">FAQ</a> | <a href="../glossary.html">Glossary</a> | <a href="../sitemap.html">Sitemap</a></p></div>