Keepalive and Healthchecks for Proxied Connections

So, I've been noticing recently that while keepalive connection pools and healthchecks are frequently used for incoming web server connections, they are used much less frequently for outbound proxied connections.

This means that proxied connections are both built up and torn down for every single proxy request, and that the proxied endpoints being used are not checked for availability or performance.

Lets fix that.

Keepalive connection pooling for Apache

Example mod_rewrite proxy invocation

RewriteRule ^(.*)$ http://www.example.com:80/$1 [P,L]

Starting with Apache 2.1 the mod_proxy ProxyPass and ProxyPassMatch directives provide implicit keepalive connection pooling

The secret sauce is described under the mod_proxy ProxyPass directive, but is appliciable to both ProxyPass and ProxyPassMatch.

Example mod_proxy ProxyPassMatch proxy invocation with implicit keepalive connection pooling

ProxyPassMatch ^(.*)$ / http://www.example.com:80/$1

Keepalive connection pooling for Nginx

Example ngx_http_upstream_module proxy invocation

upstream example {
    server www.example.com;
}

server {
    location / {
        proxy_pass http://example;
    }
}

Starting with Nginx 1.1.4 the ngx_http_upstream_module keepalive directive can be added for explicit keepalive connection pooling.

Note that for HTTP, the proxy_http_version directive should be set to “1.1” and the “Connection” header field should be cleared.

Example ngx_http_upstream_module proxy invocation with explicit keepalive connection pooling

upstream example {
    server www.example.com;
    keepalive 42;
}

server {
    location / {
        proxy_pass http://example;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

Healthchecking proxy destinations for Apache

Well, this is embarrassing. Apparently mod_proxy and mod_proxy_balancer don't actually support an out of band healthcheck mechanism.

Suck.

The problem of course is that if any of the servers are not responsive Apache will have to wait for (ProxyPass or ProxyPassMatch) -> connectiontimeout seconds to expire before timing out. connectiontimeout defaults to the Apache TimeOut value by default, which itself defaults to 300 seconds.

Apache mod_proxy will only disable a non-responsive proxy destination if error state is set for the connection, and error state is currently only set on a http 500 or 503 response

Per https://github.com/apache/httpd/blob/trunk/modules/proxy/mod_proxy.c

if (access_status == OK)
    break;  
else if (access_status == HTTP_INTERNAL_SERVER_ERROR) {
    /* Unrecoverable server error.
     * We can not failover to another worker. 
     * Mark the worker as unusable if member of load balancer
     */      
    if (balancer) {
        worker->s->status |= PROXY_WORKER_IN_ERROR;
        worker->s->error_time = apr_time_now();
    }       
    break;  
}       
else if (access_status == HTTP_SERVICE_UNAVAILABLE) {
    /* Recoverable server error.
     * We can failover to another worker
     * Mark the worker as unusable if member of load balancer
     */      
    if (balancer) {
        worker->s->status |= PROXY_WORKER_IN_ERROR;
        worker->s->error_time = apr_time_now();
    }       
}       
else {  
    /* Unrecoverable error.
     * Return the origin status code to the client. 
     */      
    break;  
} 

On the positive side, once an error state has been set, the server will be failed out for (ProxyPass or ProxyPassMatch) -> retry seconds. 60 seconds by default.

On the negative side, as no preemptive healthchecking is done, each non-responsive server will be returned to service every retry seconds until an error state is again set.

i.e. ongoing intermittent failures.

Healthchecking proxy destinations for Nginx

Example ngx_http_upstream_module proxy invocation

upstream example {
    server www1.example.com;
    server www2.example.com;
}

server {
    location / {
        proxy_pass http://example;
    }
}

The problem of course is that if any of the servers are not responsive nginx will have to wait for the server fail_timeout seconds to expire before trying another resource. By default, fail_timeout is set to 10 seconds.

On the positive side, once a server has hit max_fails, 1 by default, the server will be failed out for fail_timeout.

On the negative side, as no preemptive healthchecking is done, each non-responsive server will be returned to service every fail_timeout seconds until it again hits it's max_fails threshold.

i.e. ongoing intermittent failures.

The Nginx ngx_http_upstream_module health_check directive can be added for healthchecks.

Example ngx_http_upstream_module proxy invocation with healthchecks

upstream example {
    server www1.example.com;
    server www2.example.com;
}

server {
    location / {
        proxy_pass http://example;
        health_check;
    }
}