michaelschiemer/docs/deployment/PRODUCTION_DEPLOYMENT.md

# Production Deployment Guide

Comprehensive guide for deploying the Custom PHP Framework to production environments with Docker, SSL/TLS, secrets management, monitoring, and high availability.

## Table of Contents

1. [Prerequisites](#prerequisites)
2. [Infrastructure Setup](#infrastructure-setup)
3. [SSL/TLS Configuration](#ssltls-configuration)
4. [Secrets Management](#secrets-management)
5. [Environment Configuration](#environment-configuration)
6. [Docker Deployment](#docker-deployment)
7. [Database Migrations](#database-migrations)
8. [Monitoring and Health Checks](#monitoring-and-health-checks)
9. [Logging Configuration](#logging-configuration)
10. [Deployment Automation](#deployment-automation)
11. [Post-Deployment](#post-deployment)
12. [Troubleshooting](#troubleshooting)

---

## Prerequisites

See [deployment-prerequisites.md](deployment-prerequisites.md) for detailed requirements.

### Quick Checklist

- [ ] **Server**: Linux server with 2+ CPU cores, 4GB+ RAM, 20GB+ disk
- [ ] **Software**: Docker 20.10+, Docker Compose 2.0+
- [ ] **Network**: Public IP, open ports 80/443, firewall configured
- [ ] **Domain**: Domain name with DNS configured
- [ ] **SSL**: Let's Encrypt or commercial SSL certificate
- [ ] **Access**: SSH access with sudo privileges

### System Requirements

**Minimum**:
- 2 CPU cores
- 4 GB RAM
- 20 GB disk space
- Ubuntu 20.04+ or similar

**Recommended**:
- 4+ CPU cores
- 8+ GB RAM
- 100+ GB SSD
- Ubuntu 22.04 LTS

---

## Infrastructure Setup

### 1. Server Preparation

```bash
# Update system
sudo apt update && sudo apt upgrade -y

# Install required packages
sudo apt install -y \
    docker.io \
    docker-compose \
    nginx \
    certbot \
    python3-certbot-nginx \
    git \
    make

# Add user to docker group
sudo usermod -aG docker $USER

# Enable Docker service
sudo systemctl enable docker
sudo systemctl start docker

# Verify installation
docker --version
docker-compose --version
```

### 2. Directory Structure

```bash
# Create production directories
sudo mkdir -p /var/www/app
sudo mkdir -p /var/log/app
sudo mkdir -p /var/lib/docker/volumes/app-data
sudo mkdir -p /etc/ssl/app
sudo mkdir -p /opt/vault

# Set permissions
sudo chown -R $USER:$USER /var/www/app
sudo chown -R www-data:www-data /var/log/app
sudo chmod 755 /var/www/app
sudo chmod 755 /var/log/app
```

### 3. Clone Repository

```bash
cd /var/www
git clone https://github.com/yourusername/app.git
cd app

# Checkout production branch
git checkout production

# Set proper permissions
chmod +x scripts/deployment/*.sh
```

---

## SSL/TLS Configuration

See [ssl-certificate-management.md](ssl-certificate-management.md) for detailed SSL configuration.

### Let's Encrypt (Recommended)

```bash
# Install Certbot
sudo apt install -y certbot python3-certbot-nginx

# Obtain certificate
sudo certbot --nginx -d yourdomain.com -d www.yourdomain.com

# Verify auto-renewal
sudo certbot renew --dry-run
```

### Manual Certificate Installation

```bash
# Copy certificate files
sudo cp your-certificate.crt /etc/ssl/app/cert.pem
sudo cp your-private-key.key /etc/ssl/app/key.pem
sudo cp your-ca-bundle.crt /etc/ssl/app/ca-bundle.pem

# Set permissions
sudo chmod 600 /etc/ssl/app/*.pem
sudo chown root:root /etc/ssl/app/*.pem
```

### SSL Health Monitoring

The framework includes automatic SSL certificate monitoring:

- Warns when certificate expires within 30 days
- Critical alert when expires within 7 days
- Health check endpoint: `/health/detailed`

---

## Secrets Management

See [vault-secrets-management.md](vault-secrets-management.md) for comprehensive Vault documentation.

### 1. Initialize Vault

```bash
# Generate master encryption key
php scripts/deployment/generate-vault-key.php

# Store key securely (use password manager or secrets service)
VAULT_ENCRYPTION_KEY="<generated-key>"

# Add to environment
echo "VAULT_ENCRYPTION_KEY=${VAULT_ENCRYPTION_KEY}" >> .env.production
```

### 2. Store Production Secrets

```php
<?php
// scripts/deployment/setup-production-secrets.php
require_once __DIR__ . '/../../vendor/autoload.php';

use App\Framework\Vault\EncryptedVault;
use App\Framework\Vault\SecretKey;
use App\Framework\Vault\SecretValue;

$vault = new EncryptedVault(
    $_ENV['VAULT_ENCRYPTION_KEY'],
    '/opt/vault/production.vault'
);

// Store database credentials
$vault->set(
    SecretKey::from('db_password'),
    SecretValue::from('your-secure-database-password')
);

// Store API keys
$vault->set(
    SecretKey::from('stripe_secret_key'),
    SecretValue::from('sk_live_...')
);

// Store encryption keys
$vault->set(
    SecretKey::from('app_key'),
    SecretValue::from(bin2hex(random_bytes(32)))
);

echo "Production secrets stored successfully\n";
```

### 3. Vault Security

```bash
# Restrict Vault file permissions
sudo chown www-data:www-data /opt/vault/production.vault
sudo chmod 600 /opt/vault/production.vault

# Verify Vault health
curl http://localhost/health/detailed | jq '.checks."Vault"'
```

---

## Environment Configuration

See [production-environment-files.md](production-environment-files.md) for all environment configurations.

### 1. Create Production Environment File

```bash
# Copy template
cp .env.example .env.production

# Edit with production values
nano .env.production
```

### 2. Essential Environment Variables

```env
# Application
APP_ENV=production
APP_DEBUG=false
APP_URL=https://yourdomain.com

# Database
DB_CONNECTION=mysql
DB_HOST=db
DB_PORT=3306
DB_DATABASE=app_production
DB_USERNAME=app_user
# DB_PASSWORD stored in Vault

# Cache
CACHE_DRIVER=redis
REDIS_HOST=redis
REDIS_PORT=6379
REDIS_PASSWORD=<strong-redis-password>

# Queue
QUEUE_DRIVER=redis
QUEUE_CONNECTION=redis

# Logging
LOG_CHANNEL=production
LOG_LEVEL=info
LOG_PATH=/var/log/app

# Session
SESSION_DRIVER=redis
SESSION_LIFETIME=120

# Security
VAULT_ENCRYPTION_KEY=<from-vault-setup>

# Mail
MAIL_MAILER=smtp
MAIL_HOST=smtp.mailgun.org
MAIL_PORT=587
MAIL_USERNAME=<mailgun-username>
# MAIL_PASSWORD stored in Vault

# Admin Access
ADMIN_ALLOWED_IPS=203.0.113.0/24,198.51.100.10
```

### 3. Environment Validation

```bash
# Validate configuration
php scripts/deployment/validate-environment.php production

# Expected output:
# ✓ All required environment variables present
# ✓ Database connection successful
# ✓ Redis connection successful
# ✓ Vault accessible
# ✓ Log directory writable
# ✓ SSL certificates valid
```

---

## Docker Deployment

### 1. Build Production Images

```bash
# Build all services
docker-compose -f docker-compose.production.yml build

# Or build individually
docker-compose -f docker-compose.production.yml build php
docker-compose -f docker-compose.production.yml build nginx
```

### 2. Start Production Services

```bash
# Start all services
docker-compose -f docker-compose.production.yml up -d

# View logs
docker-compose -f docker-compose.production.yml logs -f

# Check service status
docker-compose -f docker-compose.production.yml ps
```

### 3. Production Docker Compose

See [docker-compose.production.yml](../../docker-compose.production.yml) for full configuration.

**Key Features**:
- Resource limits (CPU, memory)
- Restart policies
- Health checks
- Network isolation
- Volume management
- Logging configuration

### 4. Service Management

```bash
# Stop services
docker-compose -f docker-compose.production.yml stop

# Restart service
docker-compose -f docker-compose.production.yml restart php

# Scale workers
docker-compose -f docker-compose.production.yml up -d --scale worker=3

# Remove containers (preserves volumes)
docker-compose -f docker-compose.production.yml down

# Remove containers AND volumes (⚠️ DATA LOSS)
docker-compose -f docker-compose.production.yml down -v
```

---

## Database Migrations

See [database-migration-strategy.md](database-migration-strategy.md) for comprehensive migration guide.

### 1. Pre-Deployment Database Backup

```bash
# Backup database
docker exec app-db mysqldump \
    -u app_user \
    -p<password> \
    app_production \
    > backup_$(date +%Y%m%d_%H%M%S).sql

# Verify backup
ls -lh backup_*.sql
```

### 2. Run Migrations

```bash
# Check migration status
docker exec app-php php console.php db:status

# Run pending migrations
docker exec app-php php console.php db:migrate

# Verify migrations
docker exec app-php php console.php db:status
```

### 3. Rollback (if needed)

```bash
# Rollback last migration
docker exec app-php php console.php db:rollback 1

# Rollback multiple migrations
docker exec app-php php console.php db:rollback 3

# Restore from backup (emergency)
docker exec -i app-db mysql \
    -u app_user \
    -p<password> \
    app_production \
    < backup_20250115_120000.sql
```

### 4. Zero-Downtime Migrations

```bash
# Use blue-green deployment script
./scripts/deployment/blue-green-deploy.sh

# Or manual zero-downtime process
./scripts/deployment/zero-downtime-migration.sh
```

---

## Monitoring and Health Checks

### 1. Health Check Endpoints

**Available Endpoints**:

- **`/health`** - Basic liveness check (200 = healthy)
- **`/health/live`** - Kubernetes liveness probe
- **`/health/ready`** - Kubernetes readiness probe
- **`/health/detailed`** - Full health report with all checks
- **`/health/summary`** - Quick summary (healthy/warning/unhealthy counts)
- **`/health/category/security`** - Security-specific checks
- **`/health/category/infrastructure`** - Infrastructure checks
- **`/health/checks`** - List all registered health checks

**Example Health Check**:
```bash
# Basic health
curl http://localhost/health

# Detailed report
curl http://localhost/health/detailed | jq

# Example response:
{
  "timestamp": "2025-01-15T10:00:00+00:00",
  "overall_status": "healthy",
  "overall_healthy": true,
  "checks": {
    "Database": {
      "status": "healthy",
      "healthy": true,
      "message": "Database connection successful",
      "response_time_ms": 2.5
    },
    "Cache": {
      "status": "healthy",
      "healthy": true,
      "message": "Redis cache operational",
      "response_time_ms": 1.2
    },
    "Vault": {
      "status": "healthy",
      "healthy": true,
      "message": "Vault encryption working",
      "response_time_ms": 3.1
    },
    "SSL Certificate": {
      "status": "warning",
      "healthy": false,
      "message": "Certificate expiring in 25 days",
      "response_time_ms": 0.5,
      "details": {
        "days_until_expiry": 25,
        "issuer": "Let's Encrypt",
        "subject": "yourdomain.com"
      }
    },
    "Queue System": {
      "status": "healthy",
      "healthy": true,
      "message": "Queue operational",
      "response_time_ms": 2.8,
      "details": {
        "queue_size": 45,
        "failed_jobs": 2
      }
    }
  },
  "response_time_ms": 12.3
}
```

### 2. Prometheus Metrics

**Metrics Endpoint**: `/metrics`

```bash
# Scrape metrics
curl http://localhost/metrics

# Example Prometheus output:
# HELP health_check_database Health check status (1=healthy, 0.5=warning, 0=unhealthy)
# TYPE health_check_database gauge
health_check_database 1

# HELP health_check_database_response_time_ms Health check response time in milliseconds
# TYPE health_check_database_response_time_ms gauge
health_check_database_response_time_ms 2.5

# HELP php_memory_usage_bytes Current PHP memory usage in bytes
# TYPE php_memory_usage_bytes gauge
php_memory_usage_bytes 47185920

# HELP queue_size Total number of jobs in queue
# TYPE queue_size gauge
queue_size 45

# HELP queue_failed_jobs Total number of failed jobs
# TYPE queue_failed_jobs counter
queue_failed_jobs 2
```

### 3. Prometheus Configuration

**prometheus.yml**:
```yaml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'app'
    static_configs:
      - targets: ['app:80']
    metrics_path: '/metrics'
```

### 4. Grafana Dashboard

Import provided Grafana dashboard: `docs/deployment/grafana-dashboard.json`

**Key Metrics**:
- Health check status per component
- Response times (p50, p95, p99)
- Memory usage and trends
- Queue depth and throughput
- Error rates
- SSL certificate expiration countdown

### 5. Alerting

**Example Alertmanager Rules**:
```yaml
groups:
  - name: application
    rules:
      - alert: HealthCheckFailed
        expr: health_check_database == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Health check failed: {{ $labels.check }}"

      - alert: HighMemoryUsage
        expr: php_memory_usage_bytes / php_memory_limit_bytes > 0.9
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "PHP memory usage above 90%"

      - alert: QueueBacklog
        expr: queue_size > 1000
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Queue backlog exceeding 1000 jobs"

      - alert: SSLCertificateExpiring
        expr: ssl_certificate_days_until_expiry < 7
        for: 1h
        labels:
          severity: critical
        annotations:
          summary: "SSL certificate expiring in {{ $value }} days"
```

---

## Logging Configuration

See [production-logging.md](production-logging.md) for comprehensive logging documentation.

### 1. Production Logging Setup

The framework uses `ProductionLogConfig` with optimized handlers:

```php
// Configured in application bootstrap
use App\Framework\Logging\ProductionLogConfig;

$logConfig = ProductionLogConfig::productionWithAggregation(
    logPath: '/var/log/app',
    requestIdGenerator: $container->get(RequestIdGenerator::class)
);
```

**Features**:
- Automatic log rotation (14-day retention)
- Buffered writes for performance
- Resilient logging with fallback
- Structured JSON logs
- Request/trace context
- Performance metrics
- Log aggregation (70-90% volume reduction)

### 2. Log Levels

```
Production:  INFO and above
Staging:     DEBUG and above
Debug Mode:  ALL levels (temporary only)
```

### 3. Log Locations

```
Primary:     /var/log/app/app.log
Fallback:    /var/log/app/fallback.log
Rotation:    /var/log/app/app.log.1.gz (compressed archives)
```

### 4. Log Monitoring

```bash
# Real-time log viewing
docker exec app-php tail -f /var/log/app/app.log

# Search logs
docker exec app-php grep "ERROR" /var/log/app/app.log | tail -20

# Check log disk usage
docker exec app-php du -sh /var/log/app

# View log health
curl http://localhost/health/detailed | jq '.checks.logging'
```

### 5. Log Aggregation (Optional)

**ELK Stack Integration**:
```yaml
# docker-compose.production.yml
filebeat:
  image: docker.elastic.co/beats/filebeat:8.0.0
  volumes:
    - /var/log/app:/var/log/app:ro
    - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
  environment:
    - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
```

---

## Deployment Automation

### 1. Automated Deployment Script

```bash
# Deploy to production
./scripts/deployment/deploy-production.sh

# With custom branch
./scripts/deployment/deploy-production.sh --branch production-v2.1.0

# Dry run (no actual changes)
./scripts/deployment/deploy-production.sh --dry-run
```

### 2. Blue-Green Deployment

```bash
# Zero-downtime deployment with blue-green strategy
./scripts/deployment/blue-green-deploy.sh

# Rollback to previous version
./scripts/deployment/blue-green-rollback.sh
```

### 3. Deployment Workflow

The deployment script performs these steps:

1. **Pre-deployment**:
   - Validate environment configuration
   - Check system requirements
   - Backup database
   - Health check on current deployment

2. **Deployment**:
   - Pull latest code
   - Build Docker images
   - Run database migrations
   - Update environment files
   - Restart services

3. **Post-deployment**:
   - Health checks
   - Smoke tests
   - Log verification
   - Monitoring validation

4. **Cleanup**:
   - Remove old Docker images
   - Clean temporary files
   - Update deployment logs

### 4. CI/CD Integration

**GitHub Actions Example**:
```yaml
name: Deploy to Production

on:
  push:
    branches: [production]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Deploy to production
        uses: appleboy/ssh-action@master
        with:
          host: ${{ secrets.PRODUCTION_HOST }}
          username: ${{ secrets.PRODUCTION_USER }}
          key: ${{ secrets.PRODUCTION_SSH_KEY }}
          script: |
            cd /var/www/app
            git pull origin production
            ./scripts/deployment/deploy-production.sh
```

---

## Post-Deployment

### 1. Verification Checklist

- [ ] **Services Running**: All Docker containers healthy
  ```bash
  docker-compose -f docker-compose.production.yml ps
  ```

- [ ] **Health Checks**: All endpoints returning healthy
  ```bash
  curl http://localhost/health/detailed | jq '.overall_healthy'
  ```

- [ ] **Database**: Migrations applied successfully
  ```bash
  docker exec app-php php console.php db:status
  ```

- [ ] **SSL**: Certificate valid and auto-renewal configured
  ```bash
  curl https://yourdomain.com/health
  sudo certbot renew --dry-run
  ```

- [ ] **Vault**: Secrets accessible
  ```bash
  curl http://localhost/health/detailed | jq '.checks.Vault'
  ```

- [ ] **Logging**: Logs writing correctly
  ```bash
  docker exec app-php tail -20 /var/log/app/app.log
  ```

- [ ] **Monitoring**: Prometheus scraping metrics
  ```bash
  curl http://localhost/metrics | head -20
  ```

- [ ] **Queue**: Workers processing jobs
  ```bash
  curl http://localhost/metrics | grep queue_size
  ```

### 2. Smoke Tests

```bash
# Run automated smoke tests
./scripts/deployment/smoke-tests.sh

# Manual smoke tests
curl -I https://yourdomain.com  # Homepage
curl https://yourdomain.com/health  # Health endpoint
curl https://yourdomain.com/api/status  # API endpoint
```

### 3. Performance Testing

```bash
# Basic load test with Apache Bench
ab -n 1000 -c 10 https://yourdomain.com/

# Load test health endpoint
ab -n 5000 -c 50 https://yourdomain.com/health/summary
```

### 4. Monitoring Setup

```bash
# Add to monitoring systems
# - Prometheus target
# - Grafana dashboard
# - Alertmanager rules
# - PagerDuty/Opsgenie integration

# Verify monitoring
curl http://prometheus:9090/api/v1/targets | jq
```

---

## Troubleshooting

### Issue: Services Won't Start

**Diagnosis**:
```bash
docker-compose -f docker-compose.production.yml ps
docker-compose -f docker-compose.production.yml logs php
docker-compose -f docker-compose.production.yml logs nginx
```

**Common Causes**:
- Port conflicts (80/443 already in use)
- Invalid environment configuration
- Missing volumes or permissions
- Docker resource limits

**Solutions**:
```bash
# Check port usage
sudo netstat -tulpn | grep :80
sudo netstat -tulpn | grep :443

# Fix permissions
sudo chown -R www-data:www-data /var/log/app
sudo chmod 755 /var/log/app

# Recreate containers
docker-compose -f docker-compose.production.yml down
docker-compose -f docker-compose.production.yml up -d
```

### Issue: Health Checks Failing

**Diagnosis**:
```bash
curl -v http://localhost/health/detailed
docker exec app-php php console.php health:check
```

**Common Causes**:
- Database connection failed
- Redis connection failed
- Vault inaccessible
- SSL certificate expired/invalid
- Queue system down

**Solutions**:
```bash
# Check database
docker exec app-db mysql -u app_user -p -e "SELECT 1"

# Check Redis
docker exec app-redis redis-cli ping

# Check Vault
ls -la /opt/vault/production.vault
docker exec app-php php -r "var_dump(file_exists('/opt/vault/production.vault'));"

# Check SSL
openssl x509 -in /etc/ssl/app/cert.pem -noout -dates
```

### Issue: High Memory Usage

**Diagnosis**:
```bash
# Check container memory
docker stats

# Check PHP memory
curl http://localhost/metrics | grep php_memory
```

**Solutions**:
```bash
# Increase PHP memory limit
# In .env.production:
MEMORY_LIMIT=512M

# Restart PHP service
docker-compose -f docker-compose.production.yml restart php

# Scale down workers if needed
docker-compose -f docker-compose.production.yml up -d --scale worker=2
```

### Issue: Database Migration Failed

**Diagnosis**:
```bash
docker exec app-php php console.php db:status
docker exec app-php tail -100 /var/log/app/app.log | grep migration
```

**Solutions**:
```bash
# Rollback failed migration
docker exec app-php php console.php db:rollback 1

# Fix migration code
# Edit migration file
# Test locally first

# Re-run migration
docker exec app-php php console.php db:migrate

# Emergency: Restore from backup
docker exec -i app-db mysql -u app_user -p app_production < backup.sql
```

### Issue: SSL Certificate Problems

**Diagnosis**:
```bash
# Check certificate validity
openssl x509 -in /etc/ssl/app/cert.pem -noout -dates

# Check certificate chain
openssl verify -CAfile /etc/ssl/app/ca-bundle.pem /etc/ssl/app/cert.pem

# Test HTTPS
curl -v https://yourdomain.com/health
```

**Solutions**:
```bash
# Renew Let's Encrypt certificate
sudo certbot renew --force-renewal

# Update certificate paths in nginx config
sudo nano /etc/nginx/sites-available/app

# Restart nginx
docker-compose -f docker-compose.production.yml restart nginx

# Check SSL health
curl http://localhost/health/detailed | jq '.checks."SSL Certificate"'
```

### Issue: Queue Backlog

**Diagnosis**:
```bash
# Check queue metrics
curl http://localhost/metrics | grep queue

# Check worker logs
docker-compose -f docker-compose.production.yml logs worker
```

**Solutions**:
```bash
# Scale up workers
docker-compose -f docker-compose.production.yml up -d --scale worker=5

# Check for stuck jobs
docker exec app-php php console.php queue:failed

# Retry failed jobs
docker exec app-php php console.php queue:retry-failed

# Clear queue (emergency)
docker exec app-redis redis-cli FLUSHDB
```

---

## Rollback Procedure

### 1. Quick Rollback

```bash
# Use rollback script
./scripts/deployment/rollback-production.sh

# Or manual rollback
cd /var/www/app
git checkout <previous-commit>
docker-compose -f docker-compose.production.yml up -d --build
```

### 2. Database Rollback

```bash
# Rollback migrations
docker exec app-php php console.php db:rollback 3

# Restore database from backup
docker exec -i app-db mysql -u app_user -p app_production < backup.sql

# Verify database state
docker exec app-php php console.php db:status
```

### 3. Blue-Green Rollback

```bash
# If using blue-green deployment
./scripts/deployment/blue-green-rollback.sh

# This switches traffic back to previous version
```

---

## Security Considerations

### 1. Firewall Configuration

```bash
# Allow HTTP/HTTPS
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

# Allow SSH (if needed)
sudo ufw allow 22/tcp

# Deny all other incoming
sudo ufw default deny incoming
sudo ufw default allow outgoing

# Enable firewall
sudo ufw enable
```

### 2. Secrets Rotation

```bash
# Rotate Vault encryption key (quarterly recommended)
./scripts/deployment/rotate-vault-key.sh

# Rotate database password
./scripts/deployment/rotate-db-password.sh

# Update API keys
./scripts/deployment/update-api-keys.sh
```

### 3. Security Scanning

```bash
# Scan Docker images
docker scan app-php:latest

# Check for vulnerabilities
composer audit

# Update dependencies
composer update --with-all-dependencies
```

---

## Maintenance

### 1. Regular Maintenance Tasks

**Daily**:
- Monitor health checks
- Review error logs
- Check disk space
- Verify backup completion

**Weekly**:
- Review security alerts
- Update dependencies (if needed)
- Performance analysis
- Log rotation verification

**Monthly**:
- Security patching
- Certificate renewal check
- Database optimization
- Backup testing

### 2. Backup Strategy

```bash
# Database backups (automated)
0 2 * * * /var/www/app/scripts/deployment/backup-database.sh

# Volume backups (automated)
0 3 * * * /var/www/app/scripts/deployment/backup-volumes.sh

# Test restore (monthly)
# Restore to staging environment and verify
```

### 3. Monitoring and Alerts

Monitor these critical metrics:

- Health check status (all components)
- Response times (p95 < 200ms)
- Error rates (< 0.1%)
- Memory usage (< 80%)
- Disk space (> 20% free)
- SSL certificate expiration (> 30 days)
- Queue depth (< 1000 jobs)

---

## Additional Resources

- [Deployment Prerequisites](deployment-prerequisites.md)
- [SSL Certificate Management](ssl-certificate-management.md)
- [Production Environment Files](production-environment-files.md)
- [Database Migration Strategy](database-migration-strategy.md)
- [Vault Secrets Management](vault-secrets-management.md)
- [Production Logging Configuration](production-logging.md)
- [Docker Compose Production](../../docker-compose.production.yml)

---

## Support

For deployment issues or questions:

1. Check [Troubleshooting](#troubleshooting) section
2. Review health checks: `curl http://localhost/health/detailed`
3. Check logs: `docker-compose logs -f`
4. Verify environment: `./scripts/deployment/validate-environment.php`

For security issues, contact: security@yourdomain.com