High-Availability Cluster on Seven Servers
Clustered MySQL and PostgreSQL, a load balanced app tier, monitoring and alerting across seven dedicated servers, all built with Ansible, for a live platform that now runs on it at millions of requests a day.
A client running a busy consumer platform had outgrown one database server and one app host, where every hiccup was an outage. They rented seven dedicated servers and wanted them turned into something that keeps going when a machine dies, with both MySQL and PostgreSQL since two applications share the platform.
I built it in three stages: the database tier, then the app tier with monitoring, then alerting, admin access and the load balancer in front. The live site stayed where it was until the cluster had proven itself, and the move over was planned so it could be undone from the load balancer in seconds.
Case Study
The Problem
The platform lived on a single database and a single app server, so any failure on either was downtime, and downtime cost the client real money. They had the hardware for something better but no cluster on it. On top of that the site was live the whole time, so nothing could touch it until the new setup had been tested properly, and the cutover had to be reversible.
The Approach
Everything is Ansible, so a server can be rebuilt or added from the playbooks and not from memory. Four database servers run MySQL Group Replication and PostgreSQL under Patroni, with a three node etcd deciding who leads. Each app server has its own HAProxy that always sends writes to the current primary and spreads reads over the replicas, so the application only ever talks to localhost and never notices a failover. Redis is split in two, a queue that never evicts and a cache that does, both watched by Sentinel. Prometheus and Grafana cover every node, a clustered Alertmanager sends one alert to email and Slack instead of two, and the admin tools sit on a separate box reached over WireGuard. Cloudflare load balances in front, and the servers only accept web traffic from Cloudflare.
The Outcome
The platform moved onto the cluster with its full data set and has run there since. A database leader going away gets a replica promoted in about two seconds with nobody touching anything, and every playbook re-runs with zero changes, which is how I know the servers still look like the code. The first evening after go-live pages got slow and the load balancer got the blame. It turned out to be optimizer statistics computed while the tables were still empty, and fixing them took the worst query from two seconds to nothing. That one got its own write-up.
“My extremely complex cluster with multiple kind of fault resistant databases, app servers, and load balancers, monitoring dashboards, notification mechanism went live within unbelievable time. His Quality, communication, professionalism, availability was phenomenal. I can definitely vouch for him for devops, and will definitely give him work again in future.”
Abhishek