Tuning Apache HTTPD MPM Worker and Event

Tuning the worker and event MPMs involves modifying the configuration file (typically /etc/httpd/conf.modules.d/00-mpm.conf or /etc/httpd/conf.d/mpm.conf in RHEL environments) to adjust the balance of processes and threads.

Core Directives for Worker and Event MPMs

The following directives control the process and thread lifecycle for both worker and event MPMs:

• StartServers: The number of child server processes created upon startup.
• MinSpareThreads: The minimum number of idle threads available to handle request spikes. If the total number of idle threads drops below this value, the parent process creates new child processes.
• MaxSpareThreads: The maximum number of idle threads. If there are more idle threads than this value, the parent process kills off the excess child processes to free up system resources.
• MaxRequestWorkers (formerly MaxClients): The strict limit on the maximum number of simultaneous requests that will be served. Any connections over this limit are queued.
• ServerLimit: Sets the max number of child processes Apache can have. This is an administrative hard limit.
• ThreadsPerChild: The number of threads created by each child process. All child processes create exactly this number of threads at startup. It controls how many child processes those request worker threads are split across.
• MaxConnectionsPerChild: The maximum number of connections each child will ever process in total. Setting this to a high value prevents long-term memory leaks without causing too much CPU churn.

Configuration Formulas and Rules

The optimal starting point for MaxRequestWorkers is 300 times the number of CPU cores.

The ServerLimit is set to 16. It only needs to be increased if the number of cores is above 16. In this case, the ServerLimit can be increased to 20.

Then ThreadsPerChild value is defined by MaxRequestWorkers / ServerLimit.

If ThreadsPerChild has a value above 64, the ThreadLimit directive must be set as well with the same value.

To ensure a valid and stable configuration, the following relationships must be maintained:

1. MaxRequestWorkers = 300 * CPU Cores
2. ServerLimit = 16 (20 if the number of cores is above 16)
3. ThreadsPerChild = MaxRequestWorkers / ServerLimit
4. ThreadLimit = ThreadsPerChild
5. MinSpareThreads = 75 (the default value)
6. MaxSpareThreads = 256 (the default value)

Tuning Example: High-Traffic Environment

Below is an example configuration block tailored for a server with 4 CPU cores and 8 GB of RAM:

<IfModule mpm_worker_module>
    StartServers             3
    MinSpareThreads          75
    MaxSpareThreads          256
    ServerLimit              16
    MaxRequestWorkers        2400
    ThreadsPerChild          150
    ThreadLimit              150
    MaxConnectionsPerChild   10000
</IfModule>

<IfModule mpm_event_module>
    StartServers             3
    MinSpareThreads          75
    MaxSpareThreads          256
    ServerLimit              16
    MaxRequestWorkers        2400
    ThreadsPerChild          150
    ThreadLimit              150
    MaxConnectionsPerChild   10000
    AsyncRequestWorkerFactor 2
</IfModule>

Event-Specific Tuning: AsyncRequestWorkerFactor

The event MPM is an evolution of worker. It passes keep-alive (idle) connections to a dedicated listener thread, freeing up worker threads to handle new incoming requests.

The AsyncRequestWorkerFactor directive (default: 2) dictates how many concurrent connections are allowed per worker thread. The maximum number of concurrent connections for the event MPM is calculated as:

Maximum Concurrent Connections = (AsyncRequestWorkerFactor + 1) * MaxRequestWorkers

Increasing AsyncRequestWorkerFactor allows the server to hold open more idle keep-alive connections without utilizing active worker threads, highly beneficial for modern web applications with static assets.

Diagnostics

To accurately assess the current MPM state and tune it accordingly, utilize the following diagnostic methods.

1. Verify Active MPM

Confirm which MPM is currently loaded into the httpd binary:

httpd -V | grep MPM

Expected Output:
Server MPM: event (or worker)

2. Enable and Monitor mod_status

The mod_status module provides real-time metrics on worker thread utilization.

Follow the instructions from the article Monitoring Apache httpd performance with mod_status

Key metrics to observe:

• BusyWorkers: Indicates how many threads are actively processing requests.
• IdleWorkers: Indicates how many threads are waiting for incoming connections. If BusyWorkers frequently reaches MaxRequestWorkers, the server is dropping or queuing connections, and limits must be increased.

3. Monitor Process and Thread Counts

Inspect the system-level process and thread counts for the httpd user:

# Count total httpd processes
ps -u apache -o pid,comm | grep httpd | wc -l

# Count total httpd threads
ps -u apache -L -o pid,tid,comm | grep httpd | wc -l

4. Review Error Logs for Limit Warnings

When the server reaches configured limits, warnings are written to the error log. Search the logs for MaxRequestWorkers or ServerLimit events:

grep -i "MaxRequestWorkers" /path/to/httpd/error_log

Examples:

• [mpm_event:error] [pid 1234:tid 5678] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
• [mpm_event:error] [pid 1234:tid 5678] AH03490: scoreboard is full, not at MaxRequestWorkers. Increase ServerLimit.