For years I ran WordPress 3.x across a number of sites in this portfolio. Not because I didn't know newer versions existed — I did. But because I never had the confidence to perform a proper post-mortem on what had actually happened when those servers were compromised. The breach happened. I patched what I could see on the surface, locked the door, and moved on. That decision accumulated years of compounding technical debt and silent risk.
This article is that post-mortem — finally written, four years later. It is also a blueprint for the weekly backup and disaster recovery system now protecting the entire 100-domain multiDomainCMS portfolio, and a concrete roadmap for upgrading it to enterprise-grade business continuity.
The Original Incident: What Actually Happened
WordPress 3.x had a well-documented and extensively exploited attack surface. The version I was running, circa 2017–2020, was vulnerable to a chain of issues that are now embarrassingly obvious in retrospect:
- Unauthenticated file upload via TimThumb — a thumbnail-generation library bundled with dozens of premium themes allowed arbitrary PHP execution via crafted image URLs.
- XML-RPC brute-force amplification — with
xmlrpc.phpopen, a single HTTP request could test 500+ credential pairs through thesystem.multicallmethod. - PHP eval-injected backdoors in wp-includes — the attacker placed a base64-encoded eval payload inside
wp-includes/functions.phpthat persisted across manual "cleanups" because I only removed visible rogue files in the theme directory. - Database credential leakage via exposed wp-config.php — a misconfigured Nginx rule served
wp-config.phpas plain text to public requests during one deployment window.
Honestly? Fear and overcommitment. I had 40+ experiments running simultaneously — lead generation scraping, a Frappe CRM integration, a telemetry stack on Node .18, and various client projects. A deep forensic dive on a compromised server felt like a week I couldn't afford. So I did what most solo operators do: I replaced the theme, changed the database password, and hoped the attacker hadn't left a persistent backdoor. They had.
What I Should Have Done (The Real Post-Mortem Protocol)
A proper incident response for a compromised WordPress installation follows a structured forensic workflow. This is what I now know — and what I would do immediately if a server were compromised today:
| Phase | Action | Tool / Command |
|---|---|---|
| 1. Isolate | Take the server offline; block all inbound traffic immediately | ufw deny in; systemctl stop nginx |
| 2. Snapshot | Create a forensic disk image before any cleanup | dd if=/dev/sda of=/mnt/forensic/disk.img bs=4M |
| 3. Audit | Find all files modified in the last 30 days and files with eval/base64 | find /var/www -mtime -30 -name "*.php" | xargs grep -l "eval(base64" |
| 4. Rotate | Rotate all secrets: DB password, wp-config salts, SSH keys, API tokens | WP CLI: wp config shuffle-salts |
| 5. Restore | Provision a clean server; restore from a known-good pre-breach backup | Backup verified via SHA-256 checksum against stored manifest |
| 6. Harden | Disable XML-RPC, restrict wp-admin to LAN CIDRs, enforce CSP headers | Nginx: location /xmlrpc.php { deny all; } |
| 7. Monitor | Deploy Loki + Grafana alerting on 4xx/5xx anomalies and file integrity tripwires | OpenTelemetry → Loki → Grafana alertmanager |
The Architecture Today: Weekly Backup & Disaster Recovery
The portfolio now runs on a completely different stack — multiDomainCMS, a custom Node.js 26 / React SSR multi-tenant platform with a dual-write MongoDB + JSON file persistence model — and it is protected by a layered backup and disaster recovery system.
Weekly Backup Tiers
- Git-tracked JSON data layer — all
data/posts/anddata/settings.jsonfiles are committed to GitHub on every content update. This provides an immutable, distributed, version-controlled backup of the entire content corpus with full diff history. - MongoDB weekly mongodump — a cron-triggered
mongodumpexports all collections to a gzip-compressed BSON archive stored in/opt/backups/mongodb/on Node .32, with a 4-week rolling retention window. - Blue/Green slot state snapshots — before every zero-downtime deployment, the active Blue slot is archived as a tar.gz bundle so any failed release can be instantly rolled back without redeployment.
- Nginx configuration versioning — all
/etc/nginx/sites-available/vhost configurations are committed to a private infrastructure repo weekly.
Zero-Downtime Blue/Green Deployment Architecture
The deploy-zero-downtime.sh script implements a port-level Blue/Green swap: one slot runs on :8081, the other on :8083, with Nginx acting as the ingress proxy that switches upstream targets after health checks pass. The warm standby is always one deploy behind, meaning rollback to any previous release is a 30-second Nginx upstream swap.
With the current Blue/Green architecture, a full rollback to the previous known-good release takes under 90 seconds. A full cold-restore from the weekly MongoDB dump and Git checkout on a new bare-metal Node .32 instance takes approximately 25 minutes — the primary bottleneck being package installation, not data restoration.
The Enterprise Business Continuity Roadmap
A weekly backup and a Blue/Green swap is a solid foundation, but it falls well short of enterprise-grade Business Continuity Planning (BCP). The following roadmap outlines the staged upgrades planned for the portfolio over the next 12 months:
Phase 1 — Automated Recovery Verification (Q4 2026)
- Backup integrity verification cron — weekly automated restore-test into an isolated sandbox environment on Node .18 with automated smoke-test assertions validating that the MongoDB restore produces a queryable database with the expected document count across all tenant domains.
- SHA-256 manifest for all backup archives — every backup bundle must ship with a signed checksum manifest stored separately so tampered backups are detectable before a restore is attempted during a real incident.
- Grafana alerting on backup staleness — a Prometheus metric tracking the last successful backup timestamp for each tier; a Grafana alert fires if any tier is >8 days old.
Phase 2 — Geographic Redundancy (Q1 2027)
- Off-site cold backup to S3-compatible object storage — weekly MongoDB dump archives replicated to a Backblaze B2 or Wasabi S3 bucket in a geographically distinct region. This decouples the backup from the Node .32 host itself — a total hardware failure does not destroy the backup.
- MongoDB replica set — promoting the MongoDB instance from standalone to a 3-node replica set (Node .32 primary, Node .18 secondary, and a lightweight arbiter on a third node or cloud VM). This provides automatic leader election with sub-30-second failover, reducing the RTO from 25 minutes to under 60 seconds for database-layer failures.
- DNS failover automation — scripted DNS record updates (via Cloudflare API) that point the portfolio's A records to a hot standby IP if the primary Node .32 becomes unreachable for >3 consecutive health-check cycles.
Phase 3 — Infrastructure-as-Code & Reproducible Provisioning (Q2 2027)
- Ansible provisioning playbooks — encode the full Node .32 and Node .18 server provisioning into idempotent Ansible playbooks: package installation, PM2 ecosystem setup, Nginx vhost generation, MongoDB configuration, SSL certificate issuance via Certbot, and OpenTelemetry agent deployment. A new bare-metal server should reach a production-ready state in under 20 minutes from a single
ansible-playbook site.ymlinvocation. - Immutable deployment artifacts — each production release is tagged and stored as a versioned tarball in the S3 bucket so any historical release can be redeployed without relying on Git history being accessible.
Phase 4 — Continuous Security Posture (Q3 2027)
- Automated dependency vulnerability scanning — a weekly
pnpm audit --jsonrun dispatched through the Job Runner on Node .18, with findings pushed to a Grafana Loki stream and a Slack/webhook alert on any high-severity CVE. - File-integrity monitoring (FIM) — a lightweight Wazuh or OSSEC agent monitoring
/opt/multiDomainCMS/for unexpected file modifications, with real-time alerts surfaced in Grafana. - Ingress-level WAF rules — an Nginx ModSecurity or Cloudflare WAF ruleset blocking the OWASP CRS Top-10, applied at the ingress gateway before traffic reaches the Node.js application layer.
- Quarterly tabletop disaster recovery drills — a documented drill protocol where a full restore from the off-site S3 backup is performed on an isolated staging machine and the RTO is formally measured and logged in
docs/dr-drill-log.md.
| Metric | Current State | Target (Q3 2027) |
|---|---|---|
| Recovery Time Objective (RTO) | ~25 min (cold restore) | ~4 min (replica failover + Ansible reprovision) |
| Recovery Point Objective (RPO) | 7 days (weekly dump) | <5 min (replica streaming replication) |
| Backup Geographic Redundancy | Single-node local | 3-region (Node .32, Node .18, S3 object storage) |
| Backup Integrity Verification | Manual / none | Weekly automated restore test + SHA-256 manifest |
| Security Vulnerability Scanning | Ad-hoc | Weekly automated CVE audit via Job Runner |
| Provisioning Reproducibility | Manual / tribal knowledge | Fully automated Ansible playbooks (<20 min) |
The Honest Lesson
Discovering the compromise was not a moment of resolution — it was a moment of forced clarity. I took most of the affected sites down rather than leave them running on infrastructure I could not trust. The full portfolio could not be resurrected until recently, when I finally had a proper backup and disaster recovery foundation in place to rebuild with confidence. The delay was a prioritization failure: I had optimized for forward velocity — shipping new experiments, integrating CRM systems, building out telemetry — at the expense of the foundational resilience needed to come back at all. Rebuilding on a hacked foundation was never an option; rebuilding without resilience guarantees would have just repeated the same mistake.
The irony is that a proper post-mortem would have taken three focused days. The compounding cost of avoiding it was measured in years of suppressed anxiety and accumulated risk across 100 production domains. That math does not work in anyone's favor.
The weekly backup now running on this portfolio is not a final destination. It is the floor — the minimum defensible posture from which the enterprise roadmap above builds upward. Every phase of that roadmap has a clear, measurable outcome: lower RPO, lower RTO, higher geographic redundancy, and automated integrity verification that removes human confidence as the last line of defense.
