michaelschiemer/docs/deployment/deployment-automation.md

Production Deployment Automation

Comprehensive guide to automated production deployment scripts for the Custom PHP Framework.

Overview

The framework includes three main automation scripts for production operations:

1. production-deploy.sh - Full deployment automation (initial, update, rollback)
2. health-check.sh - Comprehensive health monitoring
3. backup.sh - Automated backup system

All scripts are located in scripts/ directory and designed for Docker-based production deployments.

Production Deployment Script

Location: scripts/production-deploy.sh

Usage

# Initial deployment (first time)
./scripts/production-deploy.sh initial

# Update deployment (zero-downtime rolling update)
./scripts/production-deploy.sh update

# Rollback to previous version
./scripts/production-deploy.sh rollback

Initial Deployment

First-time production setup with complete environment initialization:

# Prerequisites:
# 1. .env.production configured with VAULT_ENCRYPTION_KEY
# 2. docker-compose.production.yml present
# 3. Server meets hardware requirements (8GB RAM, 4 CPUs)

./scripts/production-deploy.sh initial

What it does:

1. ✅ Checks prerequisites (Docker, Compose, configuration files)
2. ✅ Verifies VAULT_ENCRYPTION_KEY is configured
3. ✅ Builds Docker images with production optimizations
4. ✅ Starts all services (web, php, db, redis, queue-worker, certbot)
5. ✅ Waits for services to be ready (20s)
6. ✅ Runs database migrations
7. ✅ Initializes SSL certificates via PHP console command
8. ✅ Verifies Vault is accessible
9. ✅ Runs health checks with retries (30 attempts)
10. ✅ Displays deployment summary

Output Example:

[12:34:56] 🚀 Starting initial production deployment...
[12:34:56] Checking prerequisites...
✅ Prerequisites check passed
[12:35:10] Building Docker images...
✅ Docker images built
[12:36:00] Starting Docker services...
[12:36:20] Running database migrations...
✅ Database migrations completed
[12:36:30] Initializing SSL certificates...
✅ SSL certificate initialized
[12:36:40] Running health checks...
✅ Health check passed

========================================
   Deployment Summary
========================================

📋 Mode: initial
⏰ Timestamp: 2024-10-25 12:37:00
📁 Project: /home/michael/dev/michaelschiemer
💾 Backup: (none - initial deployment)

🐳 Docker Services:
NAME                    STATUS
web                     Up (healthy)
php                     Up (healthy)
db                      Up (healthy)
redis                   Up (healthy)
queue-worker            Up (2 replicas)
certbot                 Up

🔒 Security Checks:
  [ ] APP_ENV=production in .env.production
  [ ] APP_DEBUG=false in .env.production
  [ ] VAULT_ENCRYPTION_KEY configured
  [ ] ADMIN_ALLOWED_IPS configured
  [ ] SSL certificates valid

✅ 🎉 Initial deployment completed successfully!

Update Deployment (Zero-Downtime)

Rolling update with automatic backup and health checks:

./scripts/production-deploy.sh update

What it does:

1. ✅ Checks prerequisites
2. ✅ Creates full backup (database, .env, storage)
3. ✅ Pulls latest images (if using registry)
4. ✅ Builds new Docker images
5. ✅ Runs database migrations
6. ✅ Rolling restart with minimal downtime:
   • PHP-FPM first (10s wait)
   • Web server next (5s wait)
   • Queue workers last (graceful shutdown via 60s grace period)
7. ✅ Runs health checks
8. ✅ Cleans up old Docker images
9. ✅ Displays summary

Zero-Downtime Strategy:

• PHP-FPM restarted first while old web server still serves requests
• Web Server restarted after PHP is ready
• Queue Workers restarted last with 60s graceful shutdown for jobs to complete
• Health checks verify each step before proceeding

Automatic Rollback on Failure: If any step fails, the script automatically rolls back to the backup:

❌ Health check failed after 30 attempts
[12:40:00] Cleaning up after error...
⚠️  Rolling back to previous version...
[12:40:10] Restoring from backup: /backups/backup_20241025_123456
✅ Database restored
✅ .env restored
✅ Storage restored
✅ Backup restored successfully

Rollback Deployment

Restore from latest backup:

./scripts/production-deploy.sh rollback

What it does:

1. ✅ Finds latest backup
2. ✅ Prompts for confirmation
3. ✅ Restores database from backup
4. ✅ Restores .env configuration
5. ✅ Restores storage directory
6. ✅ Restarts all services
7. ✅ Runs health checks

Interactive Confirmation:

⏪ Starting rollback...
⚠️  Rolling back to: /backups/backup_20241025_123456
Continue? (yes/no): yes
[12:45:00] Restoring from backup...
✅ Database restored
✅ .env restored
✅ Storage restored
✅ Backup restored successfully
✅ Health check passed
✅ 🎉 Rollback completed successfully!

Backup Strategy

Backups are created automatically during update deployments:

Backup Location: ../backups/backup_YYYYMMDD_HHMMSS_*

Backup Contents:

• backup_YYYYMMDD_HHMMSS_database.sql.gz - PostgreSQL database dump
• backup_YYYYMMDD_HHMMSS_env - .env configuration
• backup_YYYYMMDD_HHMMSS_storage.tar.gz - Storage directory (logs, cache, queue)

Retention Policy: Last 5 backups are retained, older backups are automatically cleaned up.

Error Handling

The script includes comprehensive error handling:

# Automatic cleanup on error
trap cleanup_on_error ERR

cleanup_on_error() {
    log "Cleaning up after error..."

    if [[ -d "$BACKUP_PATH" ]]; then
        warning "Rolling back to previous version..."
        restore_backup "$BACKUP_PATH"
    fi
}

Common Errors:

1. Missing VAULT_ENCRYPTION_KEY:

❌ VAULT_ENCRYPTION_KEY not configured in .env.production

Fix: Generate key with docker exec php php console.php vault:generate-key

2. Prerequisites not met:

❌ Docker is not installed

Fix: Install Docker and Docker Compose

3. Health check failed:

❌ Health check failed after 30 attempts

Fix: Check logs with docker compose logs -f --tail=100

---

Health Check Script

Location: scripts/health-check.sh

Usage

# Basic health check
./scripts/health-check.sh

# Verbose output
./scripts/health-check.sh --verbose

# JSON output (for monitoring systems)
./scripts/health-check.sh --json

Health Check Components

The script performs 12 comprehensive health checks:

1. Docker Daemon - Verifies Docker is running
2. Docker Services - Checks all 5 services (web, php, db, redis, queue-worker)
3. Web Response - HTTP response from Nginx (3 retries)
4. Health Endpoint - /health endpoint availability
5. Database - PostgreSQL connectivity via pg_isready
6. Redis - Redis ping command
7. SSL Certificate - Certificate validity via PHP console
8. Vault - Vault accessibility check
9. Disk Space - Disk usage monitoring (warn >80%, critical >90%)
10. Memory - System memory usage (warn >80%, critical >90%)
11. Queue Workers - Verify 2 workers running
12. Recent Errors - Log analysis for error frequency

Output Format

Standard Output:

[12:50:00] 🔍 Starting production health check...

[12:50:01] Checking Docker daemon...
✅ Docker daemon is running
[12:50:02] Checking Docker Compose services...
✅ All Docker services are running
[12:50:03] Checking web server response...
✅ Web server is responding
[12:50:04] Checking /health endpoint...
✅ Health endpoint is responding
[12:50:05] Checking database connectivity...
✅ Database is accepting connections
[12:50:06] Checking Redis connectivity...
✅ Redis is responding
[12:50:07] Checking SSL certificate...
✅ SSL certificate is valid
[12:50:08] Checking Vault connectivity...
✅ Vault is accessible
[12:50:09] Checking disk space...
✅ Disk space usage: 45%
[12:50:10] Checking memory usage...
✅ Memory usage: 62%
[12:50:11] Checking queue workers...
✅ Queue workers: 2 running
[12:50:12] Checking recent errors in logs...
✅ Recent errors: 2 (last 1000 lines)

========================================
   Production Health Check Summary
========================================

📊 Health Status:
  ✅ Healthy: 12
  ⚠️  Warnings: 0
  ❌ Unhealthy: 0

Overall Status: HEALTHY ✅

🎉 All critical systems are operational

========================================

JSON Output (for monitoring systems):

{
  "timestamp": "2024-10-25T12:50:12+00:00",
  "overall_status": "healthy",
  "checks": {
    "docker": "healthy",
    "service_web": "healthy",
    "service_php": "healthy",
    "service_db": "healthy",
    "service_redis": "healthy",
    "service_queue-worker": "healthy",
    "web_response": "healthy",
    "health_endpoint": "healthy",
    "database": "healthy",
    "redis": "healthy",
    "ssl": "healthy",
    "vault": "healthy",
    "disk_space": "healthy",
    "memory": "healthy",
    "queue_workers": "healthy",
    "recent_errors": "healthy"
  }
}

Verbose Mode

Provides additional details:

./scripts/health-check.sh --verbose

Additional Information:

• Active database connections
• Redis memory usage
• Full SSL certificate status
• Detailed error log excerpt

Exit Codes

• 0 - All checks healthy
• 1 - One or more critical checks failed

Integration with Monitoring

Cron Job (every 5 minutes):

*/5 * * * * /path/to/scripts/health-check.sh --json > /var/log/health-check.json

Alerting Integration:

# Send alert if unhealthy
if ! ./scripts/health-check.sh &>/dev/null; then
    ./scripts/send-alert.sh "Production health check failed"
fi

---

Backup Script

Location: scripts/backup.sh

Usage

# Full backup (database + vault + files)
./scripts/backup.sh --full

# Database only
./scripts/backup.sh --database-only

# Vault only
./scripts/backup.sh --vault-only

# Encrypted backup (GPG)
./scripts/backup.sh --full --encrypt

Backup Components

1. Database Backup

   • PostgreSQL dump via pg_dump
   • Gzipped for compression
   • Optional GPG encryption

2. Vault Backup

   • Vault secrets table (vault_secrets)
   • Vault audit table (vault_audit)
   • Encryption highly recommended

3. Environment Configuration

   • .env.production file backup
   • Contains sensitive configuration

4. Storage Directory

   • Logs, cache, queue, discovery, uploads
   • Tar.gz compression

5. Uploaded Files

   • public/uploads/ directory
   • Media and user-uploaded content

Backup Process

./scripts/backup.sh --full --encrypt

Output:

[13:00:00] 🔐 Starting production backup (type: full)...

[13:00:01] Preparing backup directory...
✅ Backup directory created: /backups/20241025_130000
[13:00:02] Backing up database...
✅ Database backup created: database.sql.gz (245M)
[13:00:05] Encrypting /backups/20241025_130000/database.sql.gz...
✅ File encrypted: database.sql.gz.gpg
[13:00:10] Backing up Vault secrets...
✅ Vault backup created: vault_secrets.sql.gz (2.3M)
⚠️  Vault backup is not encrypted - consider using --encrypt
[13:00:12] Backing up environment configuration...
✅ Environment configuration backed up
[13:00:13] Backing up storage directory...
✅ Storage backup created: storage.tar.gz (120M)
[13:00:18] Backing up uploaded files...
✅ Uploads backup created: uploads.tar.gz (1.5G)
[13:00:45] Creating backup manifest...
✅ Backup manifest created
[13:00:46] Verifying backup integrity...
✓ database.sql.gz.gpg is valid
✓ storage.tar.gz is valid
✓ uploads.tar.gz is valid
✅ All backup files verified successfully
[13:00:47] Cleaning up old backups...
✅ Old backups cleaned up (kept last 7 days)

========================================
   Backup Summary
========================================

📋 Backup Type: full
⏰ Timestamp: 2024-10-25 13:00:47
📁 Location: /backups/20241025_130000
🔒 Encrypted: true

📦 Backup Contents:

1.5G  uploads.tar.gz
245M  database.sql.gz.gpg
120M  storage.tar.gz
2.3M  vault_secrets.sql.gz

💾 Total Size: 1.9G

📝 Restoration Commands:

  Database:
    gpg -d database.sql.gz.gpg | gunzip | docker compose exec -T db psql -U postgres michaelschiemer_prod

  Vault:
    gunzip -c vault_secrets.sql.gz | docker compose exec -T db psql -U postgres michaelschiemer_prod

  Storage:
    tar -xzf storage.tar.gz -C /path/to/project

========================================
✅ 🎉 Backup completed successfully!

Backup Encryption

GPG symmetric encryption (AES-256):

./scripts/backup.sh --full --encrypt

What gets encrypted:

• Database dumps
• Vault backups
• Environment configuration

Decryption:

# Decrypt file
gpg -d database.sql.gz.gpg > database.sql.gz

# Restore database
gunzip -c database.sql.gz | docker compose exec -T db psql -U postgres michaelschiemer_prod

Encryption Password:

• Prompted during backup
• Store securely in password manager
• Required for restoration

Backup Retention

Automatic Cleanup:

• Backups older than 7 days are automatically deleted
• Keeps last 7 days of backups
• Configurable in script

Manual Cleanup:

# Remove specific backup
rm -rf /backups/20241025_130000

# Remove all backups older than 30 days
find /backups -type d -name "20*" -mtime +30 -exec rm -rf {} \;

Backup Verification

All backups are automatically verified:

Verification Checks:

• Gzip integrity (gzip -t)
• Tar.gz integrity (tar -tzf)
• File completeness

Failed Verification:

✗ database.sql.gz is corrupted
❌ Some backup files are corrupted

Restoration Procedures

Full System Restoration:

1. Restore Database:

cd /backups/20241025_130000

# If encrypted
gpg -d database.sql.gz.gpg | gunzip | docker compose exec -T db psql -U postgres michaelschiemer_prod

# If not encrypted
gunzip -c database.sql.gz | docker compose exec -T db psql -U postgres michaelschiemer_prod

2. Restore Vault:

gunzip -c vault_secrets.sql.gz | docker compose exec -T db psql -U postgres michaelschiemer_prod

3. Restore Environment:

cp env.production /path/to/project/.env.production

4. Restore Storage:

tar -xzf storage.tar.gz -C /path/to/project

5. Restore Uploads:

tar -xzf uploads.tar.gz -C /path/to/project/public

6. Restart Services:

docker compose -f docker-compose.yml -f docker-compose.production.yml --env-file .env.production restart

Automated Backup Schedule

Recommended Cron Jobs:

# Daily full backup at 2 AM (encrypted)
0 2 * * * /path/to/scripts/backup.sh --full --encrypt >> /var/log/backup.log 2>&1

# Hourly database backup
0 * * * * /path/to/scripts/backup.sh --database-only >> /var/log/backup.log 2>&1

# Weekly Vault backup (encrypted)
0 3 * * 0 /path/to/scripts/backup.sh --vault-only --encrypt >> /var/log/backup.log 2>&1

Backup Monitoring:

# Check if backup succeeded
if ! tail -1 /var/log/backup.log | grep -q "completed successfully"; then
    ./scripts/send-alert.sh "Backup failed"
fi

---

Integration with Production Workflow

Complete Production Deployment Workflow

Step 1: Initial Setup

# Prerequisites
1. Configure .env.production with all required values
2. Generate VAULT_ENCRYPTION_KEY: docker exec php php console.php vault:generate-key
3. Update ADMIN_ALLOWED_IPS for IP-based access control
4. Configure SSL_DOMAIN and SSL_EMAIL for Let's Encrypt

# Deploy
./scripts/production-deploy.sh initial

Step 2: Verify Deployment

# Run health check
./scripts/health-check.sh --verbose

# Check logs
docker compose logs -f --tail=100

# Test application
curl -H "User-Agent: Mozilla/5.0" https://your-domain.com/health

Step 3: Create Initial Backup

# Full encrypted backup
./scripts/backup.sh --full --encrypt

Step 4: Regular Updates

# Zero-downtime update
./scripts/production-deploy.sh update

# Verify health
./scripts/health-check.sh

Automated Operations

Daily Operations Cron:

# Health check every 5 minutes
*/5 * * * * /path/to/scripts/health-check.sh --json > /var/log/health-check.json

# Full backup daily at 2 AM
0 2 * * * /path/to/scripts/backup.sh --full --encrypt >> /var/log/backup.log 2>&1

# Cleanup old logs at 3 AM
0 3 * * * find /path/to/project/storage/logs -name "*.log" -mtime +30 -delete

Monitoring Integration

Prometheus Metrics (from health-check.sh JSON output):

# prometheus.yml
scrape_configs:
  - job_name: 'health-check'
    static_configs:
      - targets: ['localhost:9090']
    metrics_path: '/metrics'
    file_sd_configs:
      - files:
        - '/var/log/health-check.json'

Grafana Dashboard:

• Service status panels
• Resource usage graphs
• Error rate trends
• SSL certificate expiry countdown

---

Troubleshooting

Deployment Script Issues

Problem: Prerequisites check fails

❌ .env.production not found

Solution: Copy .env.example to .env.production and configure all values

Problem: Health check fails

❌ Health check failed after 30 attempts

Solutions:

1. Check Docker logs: docker compose logs -f php
2. Verify all services are up: docker compose ps
3. Check firewall: sudo ufw status
4. Verify DNS: dig your-domain.com

Problem: SSL initialization fails

❌ SSL initialization failed

Solutions:

1. Verify DNS A record points to server
2. Check ports 80/443 are open
3. Verify SSL_DOMAIN in .env.production
4. Check Certbot logs: docker compose logs certbot

Health Check Script Issues

Problem: Web response check fails

❌ Web server is not responding

Solutions:

1. Check Nginx status: docker compose ps web
2. Check Nginx logs: docker compose logs web
3. Verify SSL certificates: docker compose exec php php console.php ssl:status

Problem: Database check fails

❌ Database is not accepting connections

Solutions:

1. Check PostgreSQL status: docker compose ps db
2. Check PostgreSQL logs: docker compose logs db
3. Verify database credentials in .env.production

Backup Script Issues

Problem: Database backup fails

❌ Database backup failed

Solutions:

1. Check database container: docker compose ps db
2. Verify database credentials
3. Check disk space: df -h

Problem: GPG encryption fails

⚠️  GPG not installed - skipping encryption

Solution: Install GPG: sudo apt install gnupg

Problem: Backup verification fails

✗ database.sql.gz is corrupted

Solutions:

1. Run backup again
2. Check disk space
3. Verify database health

---

Best Practices

Deployment Best Practices

1. Always test in staging first
2. Run health check before deployment
3. Create backup before updates
4. Monitor logs during deployment
5. Verify health after deployment
6. Have rollback plan ready
7. Document all configuration changes

Backup Best Practices

1. Encrypt Vault backups (always)
2. Store backups off-site (AWS S3, external server)
3. Test restoration regularly (monthly)
4. Verify backup integrity (automated)
5. Monitor backup size growth
6. Rotate encryption passwords (quarterly)
7. Keep multiple backup copies (3-2-1 rule: 3 copies, 2 media types, 1 off-site)

Monitoring Best Practices

1. Run health checks every 5 minutes
2. Alert on critical failures (email, Slack, PagerDuty)
3. Monitor resource usage trends
4. Track deployment success rate
5. Review logs daily
6. Set up uptime monitoring (UptimeRobot, Pingdom)
7. Document incident responses

---

See Also

• Prerequisites: docs/deployment/production-prerequisites.md
• Environment Configuration: docs/deployment/env-production-template.md
• Docker Compose Production: docs/deployment/docker-compose-production.md
• Database Migrations: docs/deployment/database-migration-strategy.md
• SSL Setup: docs/deployment/ssl-setup.md
• Secrets Management: docs/deployment/secrets-management.md
• Logging Configuration: docs/deployment/logging-configuration.md