The “f” in “HAProxy” stands for “fun”
Today I figured I’d write this up since I’ve had to find this HAProxy config stuff a few times now and it’s always been a trial and error sort of thing - the last article I read on enabling HAProxy stats literally said it “100% works” in the title, published in 2022, annnnd - did not work. So here’s my own version that probably won’t work in a month or two.
In this article I’ll share some config on how to enable some extra functions in HAProxy to provide stats for the load balancer, an administrative socket for local operations against the live HAProxy instances, and how to use that socket to clear out Stick Tables, disable servers in backends, etc.
Not going to emphasize this part since it’s pretty easy - it’s a package called
haproxy. Install it if you haven’t already - alternatively you can run the container image instead and mount things if you like an extra layer of headaches.
# Install HAProxy, EL dnf install haproxy -y # Install HAPRoxy, Deb apt install haproxy -y # Set some basic config in /etc/haproxy/haproxy.cfg # Enable HAProxy systemctl enable --now haproxy
Configuring the Statistics Endpoint
In case you want some information on backends, frontends, servers on the backends, requests to the frontends, responses to the clients, status of the servers, etc - then you can enable the statistics endpoint for the server.
There are a few conflicting ways on how to do it that you’ll see online - ultimately, you just need to set a listener for the port you want the statistics to be served on (default is 1936), and set some
stats specific configuration there:
# global config up here # default config also maybe up here listen stats # bind sets the port and interface/IP bind *:1936 # HTTP/L7 mode is required mode http # Assume the global log settings log global # There probably aren't that many things that are supposed to be listening for maxconn maxconn 10 # Set the required timeouts timeout connect 4s timeout client 20s timeout server 20s # enable the stats module stats enable # Obfuscate the version stats hide-version # Refresh interval for the HTTP site stats refresh 30s # Enable reporting of a host name on the statistics page stats show-node # Enable reporting additional information on the statistics page stats show-legends # Set some basic user:pass auth stats auth notadmin:securePassword stats auth othernotadmin:3xt4aSecurePassword # Configure where the HTTP URL should serve from stats uri /haproxy?stats # backend/frontend config down here maybe
You can read more about all the other
stats configuration parameters here, though that should be a really good starting point. You’ve got the basic needed settings, enabling of the stats module, and some good fundamentals for keeping it secure. You should be able to do a
curl -u 'notadmin:securePassword' http://localhost:1936/haproxy?stats and see some output.
Enabling an Administrative Socket
You could serve the statistics endpoint via a system socket and interact with it via
socat - but that’s not useful since your stats may be scraped by something like your observability tool, which is why it’s available via HTTP.
However, what is very useful is enabling an administrative system socket to interact with the HAProxy instance to disable/enable frontends, healthchecks, servers, or to clear stick tables.
To enable the administrative system socket, just drop it into your
global section - alternatively, you could also place it on a
frontend to make more atomic administrative sockets:
# Maybe your global section looks like this... global log 127.0.0.1 local2 debug chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 40000 user haproxy group haproxy daemon # Enable a global administrative socket stats socket /run/haproxy/admin.sock mode 660 level admin stats timeout 10s
The key part of the
global section are the last two lines - again, you could put this in a
frontend instead for more atomic control of load balanced services and their administrative sockets.
Wherever you do put the socket, make sure to create the parent directory and properly chown/chmod it:
# Create the runtime HAProxy directory sudo mkdir -p /run/haproxy/ # Give it the proper ownership and permissions sudo chown root:root /run/haproxy/ sudo chmod 775 /run/haproxy/
Using the Administrative Socket
Now that there’s an administrative socket enabled, you can use it with something like
socat to interact with it.
You can query the socket endpoint like so:
# Get help information echo "help" | sudo socat stdio /run/haproxy/admin.sock # Get load balancer information and statistics echo "show info;show stat" | sudo socat stdio /run/haproxy/admin.sock
You can read more about the Runtime API here: https://www.haproxy.com/documentation/hapee/latest/api/runtime-api/
Clearing Stick Tables
In case you’re using a fail-over pattern that directs traffic to a primary server, failing over to a secondary and keeping the traffic there until manually reset, your
backend may look something like this:
backend https stick-table type ip size 2 nopurge stick on dst server cloudserver cloud_server_endpoint:80 check on-error mark-down observe layer7 error-limit 1 server localserver local_server_endpoint:80 check backup
In this example, your stick-table name would be the same as the backend so to clear it and reset the traffic back to the cloudserver, you’d run:
# Clear the https backend stick table echo "clear table https" | sudo socat stdio /run/haproxy/admin.sock
Disable a Backend Server
If you’re needing to test traffic patterns resolving to different servers in a backend list, you can simply disable the servers:
# Disable the cloudserver from the https backend example above echo "disable server https/cloudserver" | sudo socat stdio /run/haproxy/admin.sock # or, enable the cloudserver from the https backend example above echo "enable server https/cloudserver" | sudo socat stdio /run/haproxy/admin.sock
Bonus - Example Repo
In case you’re in the market for a reference repo that handles all this HAProxy stuff, automates it with Ansible, and with some keepalived goodness for HA HAProxy, then look no further: https://github.com/kenmoini/ansible-ha-ha-haproxy