1. Disaster Recovery Foundations: Defining RPO & RTO

Before writing a single line of shell script, an infrastructure architect must define two vital operational metrics with business stakeholders:

• Recovery Point Objective (RPO): The maximum acceptable age of data loss in the event of a disaster. If your backup runs once a week, your RPO is 7 days—meaning a crash on Wednesday afternoon permanently destroys up to 6 days of customer transactions, comments, and uploads. A modern web platform typically targets an RPO of 1 hour to 24 hours.
• Recovery Time Objective (RTO): The maximum acceptable duration of downtime required to fully restore services from cold storage. An automated restoration runbook that executes in 15 minutes achieves an RTO of 15 minutes; a complex manual recovery requiring manual server compilation may have an RTO of 48 hours.

2. The Grandfather-Father-Son (GFS) Rotation Architecture

A common rookie mistake is writing a backup script that simply overwrites a single archive file (backup.tgz) every night. If a database corruption, silent malware injection, or file truncation occurs at 1:00 AM, the 2:00 AM backup dutifully overwrites your last healthy backup with the corrupted state!

The Grandfather-Father-Son (GFS) or rolling multi-slot rotation pattern prevents this catastrophe:

• Rolling Weekly Slots (Weeks 1 through 4): Rotating across 4 separate weekly folders guarantees that you always have up to 28 days of historical recovery depth. If you discover a hidden hack that occurred three weeks ago, Week 4 or Week 3 preserves the uncompromised original code.
• Modulo Rotation Arithmetic: By storing a simple numerical counter (count) and incrementing it using modulo arithmetic (value=$(( (value % 4) + 1 ))), the script cycles gracefully without manual intervention.

3. Hardening the Bash Backup Pipeline

A naive shell script can fail silently, leave zero-byte corrupt archives, or expose root database credentials in system process listings. Below is an enterprise-hardened production backup script:

#!/usr/bin/env bash
# /opt/backups/backup.sh - Hardened Rotating Disaster Recovery Pipeline
set -euo pipefail

BACKUP_ROOT="/opt/backups"
WEB_ROOT="/var/www"
ROTATION_FILE="${BACKUP_ROOT}/count"
CNF_FILE="/root/.my.cnf"

# 1. Initialize rotation counter safely
if [ ! -f "${ROTATION_FILE}" ]; then
    echo "1" > "${ROTATION_FILE}"
fi

CURRENT_COUNT=$(cat "${ROTATION_FILE}")
SLOT=$(( (CURRENT_COUNT % 4) + 1 ))
TARGET_DIR="${BACKUP_ROOT}/week${SLOT}"
mkdir -p "${TARGET_DIR}"

echo "[$(date -Iseconds)] Starting backup for slot: week${SLOT}..."

# 2. Database Backup: Non-blocking InnoDB transaction, credentials hidden from ps
mysqldump --defaults-extra-file="${CNF_FILE}" \
    --all-databases \
    --single-transaction \
    --quick \
    --routines \
    --triggers | gzip -9 > "${TARGET_DIR}/database.sql.gz.tmp"

# Verify gzip archive integrity before promoting
gzip -t "${TARGET_DIR}/database.sql.gz.tmp"
mv "${TARGET_DIR}/database.sql.gz.tmp" "${TARGET_DIR}/database.sql.gz"

# 3. Web Asset Backup: Atomic archive creation
tar -czf "${TARGET_DIR}/www.tar.gz.tmp" -C "${WEB_ROOT}" .
mv "${TARGET_DIR}/www.tar.gz.tmp" "${TARGET_DIR}/www.tar.gz"

# 4. Generate SHA256 checksum manifest
cd "${TARGET_DIR}" && sha256sum database.sql.gz www.tar.gz > checksums.sha256

# 5. Advance rotation counter
echo "${SLOT}" > "${ROTATION_FILE}"
echo "[$(date -Iseconds)] Backup complete for week${SLOT}."

4. Scheduling via Cron & Cron Environment Traps

Scheduling the backup via crontab -e requires understanding cron execution quirks:

• Cron Minimal Environment: Cron does not execute your shell profile (.bashrc) and has a severely restricted $PATH (often just /usr/bin:/bin). Always use absolute paths (e.g., /opt/backups/backup.sh) and declare PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin at the top of your crontab.
• Standard Cron Schedule:

  # Run every Thursday at 02:00 AM server time
  0 2 * * 4 /opt/backups/backup.sh >> /var/log/backup.log 2>&1

5. Testing Disaster Recovery: The Cardinal Rule

The cardinal rule of site reliability engineering is: An untested backup is purely a hypothesis, not a backup. Periodically execute a Disaster Recovery Rehearsal on an isolated staging server:

1. Provision a fresh, empty VM running Linux, Nginx, and MariaDB.
2. Download the latest database.sql.gz and www.tar.gz archives.
3. Validate checksums: sha256sum -c checksums.sha256.
4. Extract web assets to /var/www/ and import database: zcat database.sql.gz | mysql -u root.
5. Verify that the website boots, sessions hydrate, and HTTPS endpoints respond with 200 OK.

6. Offsite Immutability: S3 Object Lock & Ransomware Immunity

Local rotating snapshots protect against software bugs and user errors, but they offer zero protection against root-level server compromise or ransomware. If an attacker gains root SSH access, their first action is executing rm -rf /opt/backups/ to destroy recovery capabilities.

To achieve true ransomware immunity, streaming backups to offsite cloud object storage with Object Lock (Write-Once-Read-Many / WORM) is mandatory:

• Compliance Mode Retention: When pushing archives to AWS S3, Cloudflare R2, or Backblaze B2, configure an Object Lock retention policy (e.g., 30 days). In compliance mode, even the root cloud account credentials cannot delete or overwrite the archive until the retention timer expires.
• Client-Side Cryptography (Restic / BorgBackup): Before transmission over the public internet, encrypt backup chunks locally using modern cryptography (AES-256-GCM or ChaCha20-Poly1305). The cloud storage provider stores purely opaque, deduplicated ciphertext blobs.
• Automated Offsite Push: Append an offsite synchronization command to the end of your backup script, ensuring local and remote repositories remain synchronized in lockstep.