michaelschiemer/docs/deployment/PRODUCTION_DEPLOYMENT.md

Production Deployment Guide

Comprehensive guide for deploying the Custom PHP Framework to production environments with Docker, SSL/TLS, secrets management, monitoring, and high availability.

Table of Contents

1. Prerequisites
2. Infrastructure Setup
3. SSL/TLS Configuration
4. Secrets Management
5. Environment Configuration
6. Docker Deployment
7. Database Migrations
8. Monitoring and Health Checks
9. Logging Configuration
10. Deployment Automation
11. Post-Deployment
12. Troubleshooting

---

Prerequisites

See deployment-prerequisites.md for detailed requirements.

Quick Checklist

• Server: Linux server with 2+ CPU cores, 4GB+ RAM, 20GB+ disk
• Software: Docker 20.10+, Docker Compose 2.0+
• Network: Public IP, open ports 80/443, firewall configured
• Domain: Domain name with DNS configured
• SSL: Let's Encrypt or commercial SSL certificate
• Access: SSH access with sudo privileges

System Requirements

Minimum:

• 2 CPU cores
• 4 GB RAM
• 20 GB disk space
• Ubuntu 20.04+ or similar

Recommended:

• 4+ CPU cores
• 8+ GB RAM
• 100+ GB SSD
• Ubuntu 22.04 LTS

---

Infrastructure Setup

1. Server Preparation

# Update system
sudo apt update && sudo apt upgrade -y

# Install required packages
sudo apt install -y \
    docker.io \
    docker-compose \
    nginx \
    certbot \
    python3-certbot-nginx \
    git \
    make

# Add user to docker group
sudo usermod -aG docker $USER

# Enable Docker service
sudo systemctl enable docker
sudo systemctl start docker

# Verify installation
docker --version
docker-compose --version

2. Directory Structure

# Create production directories
sudo mkdir -p /var/www/app
sudo mkdir -p /var/log/app
sudo mkdir -p /var/lib/docker/volumes/app-data
sudo mkdir -p /etc/ssl/app
sudo mkdir -p /opt/vault

# Set permissions
sudo chown -R $USER:$USER /var/www/app
sudo chown -R www-data:www-data /var/log/app
sudo chmod 755 /var/www/app
sudo chmod 755 /var/log/app

3. Clone Repository

cd /var/www
git clone https://github.com/yourusername/app.git
cd app

# Checkout production branch
git checkout production

# Set proper permissions
chmod +x scripts/deployment/*.sh

---

SSL/TLS Configuration

See ssl-certificate-management.md for detailed SSL configuration.

Let's Encrypt (Recommended)

# Install Certbot
sudo apt install -y certbot python3-certbot-nginx

# Obtain certificate
sudo certbot --nginx -d yourdomain.com -d www.yourdomain.com

# Verify auto-renewal
sudo certbot renew --dry-run

Manual Certificate Installation

# Copy certificate files
sudo cp your-certificate.crt /etc/ssl/app/cert.pem
sudo cp your-private-key.key /etc/ssl/app/key.pem
sudo cp your-ca-bundle.crt /etc/ssl/app/ca-bundle.pem

# Set permissions
sudo chmod 600 /etc/ssl/app/*.pem
sudo chown root:root /etc/ssl/app/*.pem

SSL Health Monitoring

The framework includes automatic SSL certificate monitoring:

• Warns when certificate expires within 30 days
• Critical alert when expires within 7 days
• Health check endpoint: /health/detailed

---

Secrets Management

See vault-secrets-management.md for comprehensive Vault documentation.

1. Initialize Vault

# Generate master encryption key
php scripts/deployment/generate-vault-key.php

# Store key securely (use password manager or secrets service)
VAULT_ENCRYPTION_KEY="<generated-key>"

# Add to environment
echo "VAULT_ENCRYPTION_KEY=${VAULT_ENCRYPTION_KEY}" >> .env.production

2. Store Production Secrets

<?php
// scripts/deployment/setup-production-secrets.php
require_once __DIR__ . '/../../vendor/autoload.php';

use App\Framework\Vault\EncryptedVault;
use App\Framework\Vault\SecretKey;
use App\Framework\Vault\SecretValue;

$vault = new EncryptedVault(
    $_ENV['VAULT_ENCRYPTION_KEY'],
    '/opt/vault/production.vault'
);

// Store database credentials
$vault->set(
    SecretKey::from('db_password'),
    SecretValue::from('your-secure-database-password')
);

// Store API keys
$vault->set(
    SecretKey::from('stripe_secret_key'),
    SecretValue::from('sk_live_...')
);

// Store encryption keys
$vault->set(
    SecretKey::from('app_key'),
    SecretValue::from(bin2hex(random_bytes(32)))
);

echo "Production secrets stored successfully\n";

3. Vault Security

# Restrict Vault file permissions
sudo chown www-data:www-data /opt/vault/production.vault
sudo chmod 600 /opt/vault/production.vault

# Verify Vault health
curl http://localhost/health/detailed | jq '.checks."Vault"'

---

Environment Configuration

See production-environment-files.md for all environment configurations.

1. Create Production Environment File

# Copy template
cp .env.example .env.production

# Edit with production values
nano .env.production

2. Essential Environment Variables

# Application
APP_ENV=production
APP_DEBUG=false
APP_URL=https://yourdomain.com

# Database
DB_CONNECTION=mysql
DB_HOST=db
DB_PORT=3306
DB_DATABASE=app_production
DB_USERNAME=app_user
# DB_PASSWORD stored in Vault

# Cache
CACHE_DRIVER=redis
REDIS_HOST=redis
REDIS_PORT=6379
REDIS_PASSWORD=<strong-redis-password>

# Queue
QUEUE_DRIVER=redis
QUEUE_CONNECTION=redis

# Logging
LOG_CHANNEL=production
LOG_LEVEL=info
LOG_PATH=/var/log/app

# Session
SESSION_DRIVER=redis
SESSION_LIFETIME=120

# Security
VAULT_ENCRYPTION_KEY=<from-vault-setup>

# Mail
MAIL_MAILER=smtp
MAIL_HOST=smtp.mailgun.org
MAIL_PORT=587
MAIL_USERNAME=<mailgun-username>
# MAIL_PASSWORD stored in Vault

# Admin Access
ADMIN_ALLOWED_IPS=203.0.113.0/24,198.51.100.10

3. Environment Validation

# Validate configuration
php scripts/deployment/validate-environment.php production

# Expected output:
# ✓ All required environment variables present
# ✓ Database connection successful
# ✓ Redis connection successful
# ✓ Vault accessible
# ✓ Log directory writable
# ✓ SSL certificates valid

---

Docker Deployment

1. Build Production Images

# Build all services
docker-compose -f docker-compose.production.yml build

# Or build individually
docker-compose -f docker-compose.production.yml build php
docker-compose -f docker-compose.production.yml build nginx

2. Start Production Services

# Start all services
docker-compose -f docker-compose.production.yml up -d

# View logs
docker-compose -f docker-compose.production.yml logs -f

# Check service status
docker-compose -f docker-compose.production.yml ps

3. Production Docker Compose

See docker-compose.production.yml for full configuration.

Key Features:

• Resource limits (CPU, memory)
• Restart policies
• Health checks
• Network isolation
• Volume management
• Logging configuration

4. Service Management

# Stop services
docker-compose -f docker-compose.production.yml stop

# Restart service
docker-compose -f docker-compose.production.yml restart php

# Scale workers
docker-compose -f docker-compose.production.yml up -d --scale worker=3

# Remove containers (preserves volumes)
docker-compose -f docker-compose.production.yml down

# Remove containers AND volumes (⚠️ DATA LOSS)
docker-compose -f docker-compose.production.yml down -v

---

Database Migrations

See database-migration-strategy.md for comprehensive migration guide.

1. Pre-Deployment Database Backup

# Backup database
docker exec app-db mysqldump \
    -u app_user \
    -p<password> \
    app_production \
    > backup_$(date +%Y%m%d_%H%M%S).sql

# Verify backup
ls -lh backup_*.sql

2. Run Migrations

# Check migration status
docker exec app-php php console.php db:status

# Run pending migrations
docker exec app-php php console.php db:migrate

# Verify migrations
docker exec app-php php console.php db:status

3. Rollback (if needed)

# Rollback last migration
docker exec app-php php console.php db:rollback 1

# Rollback multiple migrations
docker exec app-php php console.php db:rollback 3

# Restore from backup (emergency)
docker exec -i app-db mysql \
    -u app_user \
    -p<password> \
    app_production \
    < backup_20250115_120000.sql

4. Zero-Downtime Migrations

# Use blue-green deployment script
./scripts/deployment/blue-green-deploy.sh

# Or manual zero-downtime process
./scripts/deployment/zero-downtime-migration.sh

---

Monitoring and Health Checks

1. Health Check Endpoints

Available Endpoints:

• /health - Basic liveness check (200 = healthy)
• /health/live - Kubernetes liveness probe
• /health/ready - Kubernetes readiness probe
• /health/detailed - Full health report with all checks
• /health/summary - Quick summary (healthy/warning/unhealthy counts)
• /health/category/security - Security-specific checks
• /health/category/infrastructure - Infrastructure checks
• /health/checks - List all registered health checks

Example Health Check:

# Basic health
curl http://localhost/health

# Detailed report
curl http://localhost/health/detailed | jq

# Example response:
{
  "timestamp": "2025-01-15T10:00:00+00:00",
  "overall_status": "healthy",
  "overall_healthy": true,
  "checks": {
    "Database": {
      "status": "healthy",
      "healthy": true,
      "message": "Database connection successful",
      "response_time_ms": 2.5
    },
    "Cache": {
      "status": "healthy",
      "healthy": true,
      "message": "Redis cache operational",
      "response_time_ms": 1.2
    },
    "Vault": {
      "status": "healthy",
      "healthy": true,
      "message": "Vault encryption working",
      "response_time_ms": 3.1
    },
    "SSL Certificate": {
      "status": "warning",
      "healthy": false,
      "message": "Certificate expiring in 25 days",
      "response_time_ms": 0.5,
      "details": {
        "days_until_expiry": 25,
        "issuer": "Let's Encrypt",
        "subject": "yourdomain.com"
      }
    },
    "Queue System": {
      "status": "healthy",
      "healthy": true,
      "message": "Queue operational",
      "response_time_ms": 2.8,
      "details": {
        "queue_size": 45,
        "failed_jobs": 2
      }
    }
  },
  "response_time_ms": 12.3
}

2. Prometheus Metrics

Metrics Endpoint: /metrics

# Scrape metrics
curl http://localhost/metrics

# Example Prometheus output:
# HELP health_check_database Health check status (1=healthy, 0.5=warning, 0=unhealthy)
# TYPE health_check_database gauge
health_check_database 1

# HELP health_check_database_response_time_ms Health check response time in milliseconds
# TYPE health_check_database_response_time_ms gauge
health_check_database_response_time_ms 2.5

# HELP php_memory_usage_bytes Current PHP memory usage in bytes
# TYPE php_memory_usage_bytes gauge
php_memory_usage_bytes 47185920

# HELP queue_size Total number of jobs in queue
# TYPE queue_size gauge
queue_size 45

# HELP queue_failed_jobs Total number of failed jobs
# TYPE queue_failed_jobs counter
queue_failed_jobs 2

3. Prometheus Configuration

prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'app'
    static_configs:
      - targets: ['app:80']
    metrics_path: '/metrics'

4. Grafana Dashboard

Import provided Grafana dashboard: docs/deployment/grafana-dashboard.json

Key Metrics:

• Health check status per component
• Response times (p50, p95, p99)
• Memory usage and trends
• Queue depth and throughput
• Error rates
• SSL certificate expiration countdown

5. Alerting

Example Alertmanager Rules:

groups:
  - name: application
    rules:
      - alert: HealthCheckFailed
        expr: health_check_database == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Health check failed: {{ $labels.check }}"

      - alert: HighMemoryUsage
        expr: php_memory_usage_bytes / php_memory_limit_bytes > 0.9
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "PHP memory usage above 90%"

      - alert: QueueBacklog
        expr: queue_size > 1000
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Queue backlog exceeding 1000 jobs"

      - alert: SSLCertificateExpiring
        expr: ssl_certificate_days_until_expiry < 7
        for: 1h
        labels:
          severity: critical
        annotations:
          summary: "SSL certificate expiring in {{ $value }} days"

---

Logging Configuration

See production-logging.md for comprehensive logging documentation.

1. Production Logging Setup

The framework uses ProductionLogConfig with optimized handlers:

// Configured in application bootstrap
use App\Framework\Logging\ProductionLogConfig;

$logConfig = ProductionLogConfig::productionWithAggregation(
    logPath: '/var/log/app',
    requestIdGenerator: $container->get(RequestIdGenerator::class)
);

Features:

• Automatic log rotation (14-day retention)
• Buffered writes for performance
• Resilient logging with fallback
• Structured JSON logs
• Request/trace context
• Performance metrics
• Log aggregation (70-90% volume reduction)

2. Log Levels

Production:  INFO and above
Staging:     DEBUG and above
Debug Mode:  ALL levels (temporary only)

3. Log Locations

Primary:     /var/log/app/app.log
Fallback:    /var/log/app/fallback.log
Rotation:    /var/log/app/app.log.1.gz (compressed archives)

4. Log Monitoring

# Real-time log viewing
docker exec app-php tail -f /var/log/app/app.log

# Search logs
docker exec app-php grep "ERROR" /var/log/app/app.log | tail -20

# Check log disk usage
docker exec app-php du -sh /var/log/app

# View log health
curl http://localhost/health/detailed | jq '.checks.logging'

5. Log Aggregation (Optional)

ELK Stack Integration:

# docker-compose.production.yml
filebeat:
  image: docker.elastic.co/beats/filebeat:8.0.0
  volumes:
    - /var/log/app:/var/log/app:ro
    - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
  environment:
    - ELASTICSEARCH_HOSTS=http://elasticsearch:9200

---

Deployment Automation

1. Automated Deployment Script

# Deploy to production
./scripts/deployment/deploy-production.sh

# With custom branch
./scripts/deployment/deploy-production.sh --branch production-v2.1.0

# Dry run (no actual changes)
./scripts/deployment/deploy-production.sh --dry-run

2. Blue-Green Deployment

# Zero-downtime deployment with blue-green strategy
./scripts/deployment/blue-green-deploy.sh

# Rollback to previous version
./scripts/deployment/blue-green-rollback.sh

3. Deployment Workflow

The deployment script performs these steps:

1. Pre-deployment:

   • Validate environment configuration
   • Check system requirements
   • Backup database
   • Health check on current deployment

2. Deployment:

   • Pull latest code
   • Build Docker images
   • Run database migrations
   • Update environment files
   • Restart services

3. Post-deployment:

   • Health checks
   • Smoke tests
   • Log verification
   • Monitoring validation

4. Cleanup:

   • Remove old Docker images
   • Clean temporary files
   • Update deployment logs

4. CI/CD Integration

GitHub Actions Example:

name: Deploy to Production

on:
  push:
    branches: [production]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Deploy to production
        uses: appleboy/ssh-action@master
        with:
          host: ${{ secrets.PRODUCTION_HOST }}
          username: ${{ secrets.PRODUCTION_USER }}
          key: ${{ secrets.PRODUCTION_SSH_KEY }}
          script: |
            cd /var/www/app
            git pull origin production
            ./scripts/deployment/deploy-production.sh

---

Post-Deployment

1. Verification Checklist

• Services Running: All Docker containers healthy

  docker-compose -f docker-compose.production.yml ps
  

• Health Checks: All endpoints returning healthy

  curl http://localhost/health/detailed | jq '.overall_healthy'
  

• Database: Migrations applied successfully

  docker exec app-php php console.php db:status
  

• SSL: Certificate valid and auto-renewal configured

  curl https://yourdomain.com/health
  sudo certbot renew --dry-run
  

• Vault: Secrets accessible

  curl http://localhost/health/detailed | jq '.checks.Vault'
  

• Logging: Logs writing correctly

  docker exec app-php tail -20 /var/log/app/app.log
  

• Monitoring: Prometheus scraping metrics

  curl http://localhost/metrics | head -20
  

• Queue: Workers processing jobs

  curl http://localhost/metrics | grep queue_size
  

2. Smoke Tests

# Run automated smoke tests
./scripts/deployment/smoke-tests.sh

# Manual smoke tests
curl -I https://yourdomain.com  # Homepage
curl https://yourdomain.com/health  # Health endpoint
curl https://yourdomain.com/api/status  # API endpoint

3. Performance Testing

# Basic load test with Apache Bench
ab -n 1000 -c 10 https://yourdomain.com/

# Load test health endpoint
ab -n 5000 -c 50 https://yourdomain.com/health/summary

4. Monitoring Setup

# Add to monitoring systems
# - Prometheus target
# - Grafana dashboard
# - Alertmanager rules
# - PagerDuty/Opsgenie integration

# Verify monitoring
curl http://prometheus:9090/api/v1/targets | jq

---

Troubleshooting

Issue: Services Won't Start

Diagnosis:

docker-compose -f docker-compose.production.yml ps
docker-compose -f docker-compose.production.yml logs php
docker-compose -f docker-compose.production.yml logs nginx

Common Causes:

• Port conflicts (80/443 already in use)
• Invalid environment configuration
• Missing volumes or permissions
• Docker resource limits

Solutions:

# Check port usage
sudo netstat -tulpn | grep :80
sudo netstat -tulpn | grep :443

# Fix permissions
sudo chown -R www-data:www-data /var/log/app
sudo chmod 755 /var/log/app

# Recreate containers
docker-compose -f docker-compose.production.yml down
docker-compose -f docker-compose.production.yml up -d

Issue: Health Checks Failing

Diagnosis:

curl -v http://localhost/health/detailed
docker exec app-php php console.php health:check

Common Causes:

• Database connection failed
• Redis connection failed
• Vault inaccessible
• SSL certificate expired/invalid
• Queue system down

Solutions:

# Check database
docker exec app-db mysql -u app_user -p -e "SELECT 1"

# Check Redis
docker exec app-redis redis-cli ping

# Check Vault
ls -la /opt/vault/production.vault
docker exec app-php php -r "var_dump(file_exists('/opt/vault/production.vault'));"

# Check SSL
openssl x509 -in /etc/ssl/app/cert.pem -noout -dates

Issue: High Memory Usage

Diagnosis:

# Check container memory
docker stats

# Check PHP memory
curl http://localhost/metrics | grep php_memory

Solutions:

# Increase PHP memory limit
# In .env.production:
MEMORY_LIMIT=512M

# Restart PHP service
docker-compose -f docker-compose.production.yml restart php

# Scale down workers if needed
docker-compose -f docker-compose.production.yml up -d --scale worker=2

Issue: Database Migration Failed

Diagnosis:

docker exec app-php php console.php db:status
docker exec app-php tail -100 /var/log/app/app.log | grep migration

Solutions:

# Rollback failed migration
docker exec app-php php console.php db:rollback 1

# Fix migration code
# Edit migration file
# Test locally first

# Re-run migration
docker exec app-php php console.php db:migrate

# Emergency: Restore from backup
docker exec -i app-db mysql -u app_user -p app_production < backup.sql

Issue: SSL Certificate Problems

Diagnosis:

# Check certificate validity
openssl x509 -in /etc/ssl/app/cert.pem -noout -dates

# Check certificate chain
openssl verify -CAfile /etc/ssl/app/ca-bundle.pem /etc/ssl/app/cert.pem

# Test HTTPS
curl -v https://yourdomain.com/health

Solutions:

# Renew Let's Encrypt certificate
sudo certbot renew --force-renewal

# Update certificate paths in nginx config
sudo nano /etc/nginx/sites-available/app

# Restart nginx
docker-compose -f docker-compose.production.yml restart nginx

# Check SSL health
curl http://localhost/health/detailed | jq '.checks."SSL Certificate"'

Issue: Queue Backlog

Diagnosis:

# Check queue metrics
curl http://localhost/metrics | grep queue

# Check worker logs
docker-compose -f docker-compose.production.yml logs worker

Solutions:

# Scale up workers
docker-compose -f docker-compose.production.yml up -d --scale worker=5

# Check for stuck jobs
docker exec app-php php console.php queue:failed

# Retry failed jobs
docker exec app-php php console.php queue:retry-failed

# Clear queue (emergency)
docker exec app-redis redis-cli FLUSHDB

---

Rollback Procedure

1. Quick Rollback

# Use rollback script
./scripts/deployment/rollback-production.sh

# Or manual rollback
cd /var/www/app
git checkout <previous-commit>
docker-compose -f docker-compose.production.yml up -d --build

2. Database Rollback

# Rollback migrations
docker exec app-php php console.php db:rollback 3

# Restore database from backup
docker exec -i app-db mysql -u app_user -p app_production < backup.sql

# Verify database state
docker exec app-php php console.php db:status

3. Blue-Green Rollback

# If using blue-green deployment
./scripts/deployment/blue-green-rollback.sh

# This switches traffic back to previous version

---

Security Considerations

1. Firewall Configuration

# Allow HTTP/HTTPS
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

# Allow SSH (if needed)
sudo ufw allow 22/tcp

# Deny all other incoming
sudo ufw default deny incoming
sudo ufw default allow outgoing

# Enable firewall
sudo ufw enable

2. Secrets Rotation

# Rotate Vault encryption key (quarterly recommended)
./scripts/deployment/rotate-vault-key.sh

# Rotate database password
./scripts/deployment/rotate-db-password.sh

# Update API keys
./scripts/deployment/update-api-keys.sh

3. Security Scanning

# Scan Docker images
docker scan app-php:latest

# Check for vulnerabilities
composer audit

# Update dependencies
composer update --with-all-dependencies

---

Maintenance

1. Regular Maintenance Tasks

Daily:

• Monitor health checks
• Review error logs
• Check disk space
• Verify backup completion

Weekly:

• Review security alerts
• Update dependencies (if needed)
• Performance analysis
• Log rotation verification

Monthly:

• Security patching
• Certificate renewal check
• Database optimization
• Backup testing

2. Backup Strategy

# Database backups (automated)
0 2 * * * /var/www/app/scripts/deployment/backup-database.sh

# Volume backups (automated)
0 3 * * * /var/www/app/scripts/deployment/backup-volumes.sh

# Test restore (monthly)
# Restore to staging environment and verify

3. Monitoring and Alerts

Monitor these critical metrics:

• Health check status (all components)
• Response times (p95 < 200ms)
• Error rates (< 0.1%)
• Memory usage (< 80%)
• Disk space (> 20% free)
• SSL certificate expiration (> 30 days)
• Queue depth (< 1000 jobs)

---

Additional Resources

• Deployment Prerequisites
• SSL Certificate Management
• Production Environment Files
• Database Migration Strategy
• Vault Secrets Management
• Production Logging Configuration
• Docker Compose Production

---

Support

For deployment issues or questions:

1. Check Troubleshooting section
2. Review health checks: curl http://localhost/health/detailed
3. Check logs: docker-compose logs -f
4. Verify environment: ./scripts/deployment/validate-environment.php

For security issues, contact: security@yourdomain.com