Configuring Hiawatha With Peacemaker for Production Use
I spent about three weeks trying to get Hiawatha running reliably in a clustered setup with Peacemaker, and honestly, the documentation doesn't really cover the messy middle part. Hiawatha is a lightweight web server designed for Unix-like systems. It uses an event-driven architecture that handles concurrent connections differently than Apache or Nginx, which matters a lot when you're putting it behind a cluster resource manager. Peacemaker is the open-source cluster resource manager that handles failover and load balancing decisions. Combining them works, but you have to understand how Hiawatha actually behaves under load before you start wiring the cluster constraints together. The basic idea is that Peacemaker manages Hiawatha as a cluster resource. When one node fails, Peacemaker detects the failure through health checks and migrates the virtual IP or service to another node. Hiawatha's event loop means the failure detection works a bit differently than with other web servers. You can't just check if the process is running. You have to verify that the server is actually accepting connections, otherwise Peacemaker might think a degraded node is healthy when it isn't. I ran into this exact problem on a production stack. The initial resource agent was checking process liveness with a simple pgrep command. After about six hours of uptime, Hiawatha on one node started dropping connections under load. The process stayed alive. Peacemaker kept running probes every 30 seconds. The degraded node continued to hold the resource while clients hit connection timeouts. I ended up writing a custom health check that connected to the Hiawatha status port and verified the response time and connection count were within acceptable bounds. The check took about 800 milliseconds to complete and caught the degradation that the process check missed. If you're building this yourself, don't skip the connection-level health check. The default ones won't catch stuck worker pools.
Installation and base configuration
Install Hiawatha from your distribution's package manager or compile from source. The package version on Debian or Ubuntu is usually functional but may be behind the latest release. If you're running a high-traffic environment, compile from source and grab the current version from the Hiawatha website. Peacemaker comes through the standard distribution repositories on RHEL, CentOS, and Debian-based systems. You'll also need corosync or pacemaker-base depending on your platform. I typically run corosync for the cluster messaging layer because it handles node communication faster than the older heartbeat framework. Configure Hiawatha before touching the cluster. Set up your virtual hosts, logging, and any CGI or FastCGI backends you need. Make sure the configuration loads cleanly by running hiawatha with the -t flag, which tests the config without starting the daemon. Check your error log after the test. Hiawatha is strict about permission issues and missing directories, so a failed config test usually points directly to a filesystem problem.
Setting up Peacemaker constraints
Once Hiawatha is working as a standalone server, you add it to the cluster. The basic resource definition uses the ocf:heartbeat:hiawatha agent if your distribution provides it. If not, you write a simple shell script that starts, stops, and monitors the service using Hiawatha's native commands. Peacemaker supports OCF agents natively, so if a packaged agent exists for your platform, use it and avoid the custom script path unless you have a reason. The constraint configuration determines how the cluster behaves. A typical setup uses a colocation constraint to keep the Hiawatha resource on the same node as your virtual IP, and a order constraint to define what starts first. I usually set the virtual IP to start before the web server resource to avoid clients connecting to a node that hasn't finished initialization. The exact timing depends on your application. For Hiawatha, which starts in under a second, this rarely matters, but if you have backend services attached, the order becomes critical. Set the resource stickiness carefully. Hiawatha maintains its own connection tracking through its event loop, so migrating the service mid-session disrupts active connections. A stickiness value between 50 and 100 prevents unnecessary failbacks while still allowing Peacemaker to rebalance after a prolonged outage. Too high and the cluster becomes slow to recover from maintenance windows. Too low and you get flapping during minor network partitions.
Get the Full Details
Health check configuration
This is where most people mess up. The default probe interval and timeout values don't work well for Hiawatha in practice. I recommend setting the monitor interval to 10 seconds with a timeout of 5 seconds. The default 30-second probe interval is too slow for detecting a stuck worker pool. When Hiawatha locks up under certain conditions, you want Peacemaker to know about it within a reasonable window so it can fail over. Use the custom health check script I mentioned earlier. Connect to the Hiawatha status page or port, verify the HTTP response code is 200, and measure response time. If the response exceeds your threshold, return a failure exit code and Peacemaker marks the node as unfit for the resource. A 3-second timeout on the health check is usually sufficient. Anything shorter risks false failures during network latency spikes. I also recommend enabling Hiawatha's built-in monitoring and statistics output. The status page gives you visibility into active connections, request rates, and memory usage. Without it, you're flying blind inside the cluster. Log the status data to a file that your monitoring system can read. This helps you distinguish between a Hiawatha problem and a Peacemaker problem when things go wrong, which happens more often than the docs suggest.
Logging and troubleshooting
Enable detailed logging in both Hiawatha and Peacemaker during your initial setup. Hiawatha's access log shows incoming requests and response times. Its error log captures configuration issues, permission errors, and runtime exceptions. Peacemaker logs go to syslog by default. Check /var/log/messages or /var/log/cluster/ depending on your distribution for resource state transitions and probe results. When you run into unexplained failovers, the first thing I check is the probe output. Run cibadmin --query to inspect the current cluster state and resource settings. cibadmin --verify checks the configuration for syntax issues. These commands don't hurt anything and save a lot of time when you're trying to figure out why Peacemaker moved the resource at 3 AM for no apparent reason. Another issue to watch for is split-brain scenarios. Make sure your quorum settings match your node count. A two-node cluster with quorum=1 will always form a valid cluster as long as both nodes are reachable, but if the network partitions, both nodes may think they're the primary. Hiawatha can handle concurrent requests on multiple nodes, but database locks and session state become problematic. Consider fencing agents or STONITH devices to prevent this, or use a three-node cluster where the majority rule works as intended.
Common pitfalls
Running Hiawatha with FastCGI backends behind Peacemaker requires additional coordination. The FastCGI child processes may not restart automatically when the node fails over. I had to configure a separate resource for the FastCGI spawner or ensure the backend process manager restarted on node promotion. Otherwise, Hiawatha would accept connections but forward them to a backend that wasn't listening. SSL certificate handling is another area that trips people up. If you're using Let's Encrypt or similar automated renewal, make sure the certificate files are accessible on all cluster nodes. Peacemaker doesn't manage file systems automatically unless you configure a shared storage resource. I solved this by mounting the certificate directory over NFS and adding a filesystem resource to the cluster. That way, both nodes always have access to the current certificates without manual synchronization. Performance-wise, Hiawatha with Peacemaker handles moderate traffic well. I've seen it sustain around 5000 concurrent connections per node without issues. Beyond that, you'll want to tune the event loop settings and consider running multiple instances or scaling horizontally. The event-driven model is efficient, but it's not magic. Kernel parameters like somaxconn and net.core.somaxconn affect how many connections can be queued before Hiawatha starts rejecting them.
Where this setup falls short
Hiawatha and Peacemaker isn't the right choice if you need deep application-level routing, complex load balancing rules, or integration with a modern API gateway. The cluster manager handles service migration and health checks. It doesn't do content-based routing or header manipulation. If your application requires that, put HAProxy or Nginx in front of the Hiawatha cluster and let Peacemaker manage only the backend tier. That separation keeps things simpler and makes troubleshooting easier. The other limitation is operational overhead. Managing a Peacemaker cluster adds complexity that small projects rarely need. If you're running a single server or don't require high availability, Hiawatha alone handles most workloads without the cluster management layer. The failover benefit only justifies the setup cost when downtime actually matters to your users. For a straightforward Hiawatha and The Peacemaker deployment, focus on getting the health checks right, leave stickiness at a moderate level, and verify your logging before you commit to production. The rest is mostly incremental tuning based on your actual traffic patterns.
