Commit 01bb7186 authored by Graham Leggett's avatar Graham Leggett
Browse files

MPMs: Support SO_REUSEPORT to create multiple duplicated listener 

records for scalability.

Submitted by: Yingqi Lu <yingqi.lu@intel.com>, Jeff Trawick,
              Jim Jagielski, Yann Ylavic

Reviewed by: ylavic, jim, minfrin


git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/branches/2.4.x@1705492 13f79535-47bb-0310-9956-ffa450edef68
parent f14ed219

CHANGES

+4 −0
Original line number Diff line number Diff line
@@ -2,6 +2,10 @@ 


Changes with Apache 2.4.17



  *) MPMs: Support SO_REUSEPORT to create multiple duplicated listener

     records for scalability. [Yingqi Lu <yingqi.lu@intel.com>,

     Jeff Trawick, Jim Jagielski, Yann Ylavic]



  *) mod_proxy: Fix a race condition that caused a failed worker to be retried

     before the retry period is over. [Ruediger Pluem]

STATUS

+0 −21
Original line number Diff line number Diff line
@@ -109,27 +109,6 @@ RELEASE SHOWSTOPPERS: 
PATCHES ACCEPTED TO BACKPORT FROM TRUNK:

  [ start all new proposals below, under PATCHES PROPOSED. ]



   * MPMs: Support SO_REUSEPORT to create multiple duplicated listener

           records for scalability (full log in 2.4.x patch).

     trunk patch: http://svn.apache.org/r1599531

                  http://svn.apache.org/r1599593

                  http://svn.apache.org/r1599601

                  http://svn.apache.org/r1599603

                  http://svn.apache.org/r1601558

                  http://svn.apache.org/r1629909

                  http://svn.apache.org/r1629918

                  http://svn.apache.org/r1629990

                  http://svn.apache.org/r1635521

                  http://svn.apache.org/r1635859

                  http://svn.apache.org/r1640145

                  http://svn.apache.org/r1640161

                  http://svn.apache.org/r1640184

                  http://svn.apache.org/r1640763

                  http://svn.apache.org/r1643179

                  http://svn.apache.org/r1656368

                  http://svn.apache.org/r1679714

     2.4.x patch: http://people.apache.org/~ylavic/httpd-2.4.x-ap_listeners_buckets-v3.patch

     +1: ylavic, jim, minfrin

 



PATCHES PROPOSED TO BACKPORT FROM TRUNK:

docs/manual/mod/mpm_common.xml

+38 −0
Original line number Diff line number Diff line
@@ -252,6 +252,44 @@ discussion of the <code>Address already in use</code> error message, 
including other causes.</a></seealso>

</directivesynopsis>



<directivesynopsis>

<name>ListenCoresBucketsRatio</name>

<description>Ratio between the number of CPU cores (online) and the number of

listeners' buckets</description>

<syntax>ListenCoresBucketsRatio <var>ratio</var></syntax>

<default>ListenCoresBucketsRatio 0 (disabled)</default>

<contextlist><context>server config</context></contextlist>

<modulelist>

<module>event</module>

<module>prefork</module>

<module>worker</module></modulelist>

<compatibility>Available in Apache HTTP Server 2.4.13, with a kernel supporting

the socket option <code>SO_REUSEPORT</code> and distributing new connections

evenly accross listening processes' (or threads') sockets using it (eg. Linux

3.9 and later, but not the current implementations of <code>SO_REUSEPORT</code>

in *BSDs.</compatibility>



<usage>

    <p>A <var>ratio</var> between the number of (online) CPU cores and the

    number of listeners' buckets can be used to make Apache HTTP Server create

    <code>num_cpu_cores / ratio</code> listening buckets, each containing its

    own <directive>Listen</directive>-ing socket(s) on the same port(s), and

    then make each child handle a single bucket (with round-robin distribution

    of the buckets at children creation time).</p>



    <p><directive>ListenCoresBucketsRatio</directive> can improve the

    scalability when accepting new connections is/becomes the bottleneck.

    On systems with a large number of CPU cores, enabling this feature has

    been tested to show significant performances improvement and shorter

    responses time.</p>



    <p>There must be at least twice the number of CPU cores than the

    configured <var>ratio</var> for this to be active. The recommended

    <var>ratio</var> is <code>8</code>, hence at least <code>16</code>

    cores should be available at runtime when this value is used.</p>

</usage>

</directivesynopsis>



<directivesynopsis>

<name>ListenBackLog</name>

<description>Maximum length of the queue of pending connections</description>

include/ap_listen.h

+28 −0
Original line number Diff line number Diff line
@@ -77,6 +77,8 @@ struct ap_listen_rec { 
 * The global list of ap_listen_rec structures

 */

AP_DECLARE_DATA extern ap_listen_rec *ap_listeners;

AP_DECLARE_DATA extern int ap_num_listen_buckets;

AP_DECLARE_DATA extern int ap_have_so_reuseport;



/**

 * Setup all of the defaults for the listener list
@@ -91,11 +93,34 @@ AP_DECLARE(void) ap_listen_pre_config(void); 
 */

AP_DECLARE(int) ap_setup_listeners(server_rec *s);



/**

 * This function duplicates ap_listeners into multiple buckets when configured

 * to (see ListenCoresBucketsRatio) and the platform supports it (eg. number of

 * online CPU cores and SO_REUSEPORT available).

 * @param p The config pool

 * @param s The global server_rec

 * @param buckets The array of listeners buckets.

 * @param num_buckets The total number of listeners buckets (array size).

 * @remark If the given *num_buckets is 0 (input), it will be computed

 *         according to the platform capacities, otherwise (positive) it

 *         will be preserved. The number of listeners duplicated will

 *         always match *num_buckets, be it computed or given.

 */

AP_DECLARE(apr_status_t) ap_duplicate_listeners(apr_pool_t *p, server_rec *s,

                                                ap_listen_rec ***buckets,

                                                int *num_buckets);



/**

 * Loop through the global ap_listen_rec list and close each of the sockets.

 */

AP_DECLARE_NONSTD(void) ap_close_listeners(void);



/**

 * Loop through the given ap_listen_rec list and close each of the sockets.

 * @param listener The listener to close.

 */

AP_DECLARE_NONSTD(void) ap_close_listeners_ex(ap_listen_rec *listeners);



/**

 * FIXMEDOC

 */
@@ -109,6 +134,7 @@ AP_DECLARE_NONSTD(int) ap_close_selected_listeners(ap_slave_t *); 
 * called.

 */

AP_DECLARE_NONSTD(const char *) ap_set_listenbacklog(cmd_parms *cmd, void *dummy, const char *arg);

AP_DECLARE_NONSTD(const char *) ap_set_listencbratio(cmd_parms *cmd, void *dummy, const char *arg);

AP_DECLARE_NONSTD(const char *) ap_set_listener(cmd_parms *cmd, void *dummy,

                                                int argc, char *const argv[]);

AP_DECLARE_NONSTD(const char *) ap_set_send_buffer_size(cmd_parms *cmd, void *dummy,
@@ -120,6 +146,8 @@ AP_DECLARE_NONSTD(const char *) ap_set_receive_buffer_size(cmd_parms *cmd, 
#define LISTEN_COMMANDS \

AP_INIT_TAKE1("ListenBacklog", ap_set_listenbacklog, NULL, RSRC_CONF, \

  "Maximum length of the queue of pending connections, as used by listen(2)"), \

AP_INIT_TAKE1("ListenCoresBucketsRatio", ap_set_listencbratio, NULL, RSRC_CONF, \

  "Ratio between the number of CPU cores (online) and the number of listeners buckets"), \

AP_INIT_TAKE_ARGV("Listen", ap_set_listener, NULL, RSRC_CONF, \

  "A port number or a numeric IP address and a port number, and an optional protocol"), \

AP_INIT_TAKE1("SendBufferSize", ap_set_send_buffer_size, NULL, RSRC_CONF, \

include/ap_mmn.h

+6 −1
Original line number Diff line number Diff line
@@ -445,6 +445,11 @@ 
 * 20120211.46 (2.4.13-dev) Add ap_map_http_request_error()

 * 20120211.47 (2.4.13-dev) Add ap_some_authn_required, ap_force_authn hook.

 *                          Deprecate broken ap_some_auth_required.

 * 20120211.48 (2.4.13-dev) Added ap_log_mpm_common().

 * 20120211.49 (2.4.13-dev) Add listener bucket in scoreboard.h's process_score.

 * 20120211.50 (2.4.13-dev) Add ap_set_listencbratio(), ap_close_listeners_ex(),

 *                          ap_duplicate_listeners(), ap_num_listen_buckets and

 *                          ap_have_so_reuseport to ap_listen.h.

 */



#define MODULE_MAGIC_COOKIE 0x41503234UL /* "AP24" */
@@ -452,7 +457,7 @@ 
#ifndef MODULE_MAGIC_NUMBER_MAJOR

#define MODULE_MAGIC_NUMBER_MAJOR 20120211

#endif

#define MODULE_MAGIC_NUMBER_MINOR 47                   /* 0...n */

#define MODULE_MAGIC_NUMBER_MINOR 50                   /* 0...n */



/**

 * Determine if the server's current MODULE_MAGIC_NUMBER is at least a