# Production Deployment Guide Comprehensive guide for deploying the Custom PHP Framework to production environments with Docker, SSL/TLS, secrets management, monitoring, and high availability. ## Table of Contents 1. [Prerequisites](#prerequisites) 2. [Infrastructure Setup](#infrastructure-setup) 3. [SSL/TLS Configuration](#ssltls-configuration) 4. [Secrets Management](#secrets-management) 5. [Environment Configuration](#environment-configuration) 6. [Docker Deployment](#docker-deployment) 7. [Database Migrations](#database-migrations) 8. [Monitoring and Health Checks](#monitoring-and-health-checks) 9. [Logging Configuration](#logging-configuration) 10. [Deployment Automation](#deployment-automation) 11. [Post-Deployment](#post-deployment) 12. [Troubleshooting](#troubleshooting) --- ## Prerequisites See [deployment-prerequisites.md](deployment-prerequisites.md) for detailed requirements. ### Quick Checklist - [ ] **Server**: Linux server with 2+ CPU cores, 4GB+ RAM, 20GB+ disk - [ ] **Software**: Docker 20.10+, Docker Compose 2.0+ - [ ] **Network**: Public IP, open ports 80/443, firewall configured - [ ] **Domain**: Domain name with DNS configured - [ ] **SSL**: Let's Encrypt or commercial SSL certificate - [ ] **Access**: SSH access with sudo privileges ### System Requirements **Minimum**: - 2 CPU cores - 4 GB RAM - 20 GB disk space - Ubuntu 20.04+ or similar **Recommended**: - 4+ CPU cores - 8+ GB RAM - 100+ GB SSD - Ubuntu 22.04 LTS --- ## Infrastructure Setup ### 1. Server Preparation ```bash # Update system sudo apt update && sudo apt upgrade -y # Install required packages sudo apt install -y \ docker.io \ docker-compose \ nginx \ certbot \ python3-certbot-nginx \ git \ make # Add user to docker group sudo usermod -aG docker $USER # Enable Docker service sudo systemctl enable docker sudo systemctl start docker # Verify installation docker --version docker-compose --version ``` ### 2. Directory Structure ```bash # Create production directories sudo mkdir -p /var/www/app sudo mkdir -p /var/log/app sudo mkdir -p /var/lib/docker/volumes/app-data sudo mkdir -p /etc/ssl/app sudo mkdir -p /opt/vault # Set permissions sudo chown -R $USER:$USER /var/www/app sudo chown -R www-data:www-data /var/log/app sudo chmod 755 /var/www/app sudo chmod 755 /var/log/app ``` ### 3. Clone Repository ```bash cd /var/www git clone https://github.com/yourusername/app.git cd app # Checkout production branch git checkout production # Set proper permissions chmod +x scripts/deployment/*.sh ``` --- ## SSL/TLS Configuration See [ssl-certificate-management.md](ssl-certificate-management.md) for detailed SSL configuration. ### Let's Encrypt (Recommended) ```bash # Install Certbot sudo apt install -y certbot python3-certbot-nginx # Obtain certificate sudo certbot --nginx -d yourdomain.com -d www.yourdomain.com # Verify auto-renewal sudo certbot renew --dry-run ``` ### Manual Certificate Installation ```bash # Copy certificate files sudo cp your-certificate.crt /etc/ssl/app/cert.pem sudo cp your-private-key.key /etc/ssl/app/key.pem sudo cp your-ca-bundle.crt /etc/ssl/app/ca-bundle.pem # Set permissions sudo chmod 600 /etc/ssl/app/*.pem sudo chown root:root /etc/ssl/app/*.pem ``` ### SSL Health Monitoring The framework includes automatic SSL certificate monitoring: - Warns when certificate expires within 30 days - Critical alert when expires within 7 days - Health check endpoint: `/health/detailed` --- ## Secrets Management See [vault-secrets-management.md](vault-secrets-management.md) for comprehensive Vault documentation. ### 1. Initialize Vault ```bash # Generate master encryption key php scripts/deployment/generate-vault-key.php # Store key securely (use password manager or secrets service) VAULT_ENCRYPTION_KEY="" # Add to environment echo "VAULT_ENCRYPTION_KEY=${VAULT_ENCRYPTION_KEY}" >> .env.production ``` ### 2. Store Production Secrets ```php set( SecretKey::from('db_password'), SecretValue::from('your-secure-database-password') ); // Store API keys $vault->set( SecretKey::from('stripe_secret_key'), SecretValue::from('sk_live_...') ); // Store encryption keys $vault->set( SecretKey::from('app_key'), SecretValue::from(bin2hex(random_bytes(32))) ); echo "Production secrets stored successfully\n"; ``` ### 3. Vault Security ```bash # Restrict Vault file permissions sudo chown www-data:www-data /opt/vault/production.vault sudo chmod 600 /opt/vault/production.vault # Verify Vault health curl http://localhost/health/detailed | jq '.checks."Vault"' ``` --- ## Environment Configuration See [production-environment-files.md](production-environment-files.md) for all environment configurations. ### 1. Create Production Environment File ```bash # Copy template cp .env.example .env.production # Edit with production values nano .env.production ``` ### 2. Essential Environment Variables ```env # Application APP_ENV=production APP_DEBUG=false APP_URL=https://yourdomain.com # Database DB_CONNECTION=mysql DB_HOST=db DB_PORT=3306 DB_DATABASE=app_production DB_USERNAME=app_user # DB_PASSWORD stored in Vault # Cache CACHE_DRIVER=redis REDIS_HOST=redis REDIS_PORT=6379 REDIS_PASSWORD= # Queue QUEUE_DRIVER=redis QUEUE_CONNECTION=redis # Logging LOG_CHANNEL=production LOG_LEVEL=info LOG_PATH=/var/log/app # Session SESSION_DRIVER=redis SESSION_LIFETIME=120 # Security VAULT_ENCRYPTION_KEY= # Mail MAIL_MAILER=smtp MAIL_HOST=smtp.mailgun.org MAIL_PORT=587 MAIL_USERNAME= # MAIL_PASSWORD stored in Vault # Admin Access ADMIN_ALLOWED_IPS=203.0.113.0/24,198.51.100.10 ``` ### 3. Environment Validation ```bash # Validate configuration php scripts/deployment/validate-environment.php production # Expected output: # ✓ All required environment variables present # ✓ Database connection successful # ✓ Redis connection successful # ✓ Vault accessible # ✓ Log directory writable # ✓ SSL certificates valid ``` --- ## Docker Deployment ### 1. Build Production Images ```bash # Build all services docker-compose -f docker-compose.production.yml build # Or build individually docker-compose -f docker-compose.production.yml build php docker-compose -f docker-compose.production.yml build nginx ``` ### 2. Start Production Services ```bash # Start all services docker-compose -f docker-compose.production.yml up -d # View logs docker-compose -f docker-compose.production.yml logs -f # Check service status docker-compose -f docker-compose.production.yml ps ``` ### 3. Production Docker Compose See [docker-compose.production.yml](../../docker-compose.production.yml) for full configuration. **Key Features**: - Resource limits (CPU, memory) - Restart policies - Health checks - Network isolation - Volume management - Logging configuration ### 4. Service Management ```bash # Stop services docker-compose -f docker-compose.production.yml stop # Restart service docker-compose -f docker-compose.production.yml restart php # Scale workers docker-compose -f docker-compose.production.yml up -d --scale worker=3 # Remove containers (preserves volumes) docker-compose -f docker-compose.production.yml down # Remove containers AND volumes (⚠️ DATA LOSS) docker-compose -f docker-compose.production.yml down -v ``` --- ## Database Migrations See [database-migration-strategy.md](database-migration-strategy.md) for comprehensive migration guide. ### 1. Pre-Deployment Database Backup ```bash # Backup database docker exec app-db mysqldump \ -u app_user \ -p \ app_production \ > backup_$(date +%Y%m%d_%H%M%S).sql # Verify backup ls -lh backup_*.sql ``` ### 2. Run Migrations ```bash # Check migration status docker exec app-php php console.php db:status # Run pending migrations docker exec app-php php console.php db:migrate # Verify migrations docker exec app-php php console.php db:status ``` ### 3. Rollback (if needed) ```bash # Rollback last migration docker exec app-php php console.php db:rollback 1 # Rollback multiple migrations docker exec app-php php console.php db:rollback 3 # Restore from backup (emergency) docker exec -i app-db mysql \ -u app_user \ -p \ app_production \ < backup_20250115_120000.sql ``` ### 4. Zero-Downtime Migrations ```bash # Use blue-green deployment script ./scripts/deployment/blue-green-deploy.sh # Or manual zero-downtime process ./scripts/deployment/zero-downtime-migration.sh ``` --- ## Monitoring and Health Checks ### 1. Health Check Endpoints **Available Endpoints**: - **`/health`** - Basic liveness check (200 = healthy) - **`/health/live`** - Kubernetes liveness probe - **`/health/ready`** - Kubernetes readiness probe - **`/health/detailed`** - Full health report with all checks - **`/health/summary`** - Quick summary (healthy/warning/unhealthy counts) - **`/health/category/security`** - Security-specific checks - **`/health/category/infrastructure`** - Infrastructure checks - **`/health/checks`** - List all registered health checks **Example Health Check**: ```bash # Basic health curl http://localhost/health # Detailed report curl http://localhost/health/detailed | jq # Example response: { "timestamp": "2025-01-15T10:00:00+00:00", "overall_status": "healthy", "overall_healthy": true, "checks": { "Database": { "status": "healthy", "healthy": true, "message": "Database connection successful", "response_time_ms": 2.5 }, "Cache": { "status": "healthy", "healthy": true, "message": "Redis cache operational", "response_time_ms": 1.2 }, "Vault": { "status": "healthy", "healthy": true, "message": "Vault encryption working", "response_time_ms": 3.1 }, "SSL Certificate": { "status": "warning", "healthy": false, "message": "Certificate expiring in 25 days", "response_time_ms": 0.5, "details": { "days_until_expiry": 25, "issuer": "Let's Encrypt", "subject": "yourdomain.com" } }, "Queue System": { "status": "healthy", "healthy": true, "message": "Queue operational", "response_time_ms": 2.8, "details": { "queue_size": 45, "failed_jobs": 2 } } }, "response_time_ms": 12.3 } ``` ### 2. Prometheus Metrics **Metrics Endpoint**: `/metrics` ```bash # Scrape metrics curl http://localhost/metrics # Example Prometheus output: # HELP health_check_database Health check status (1=healthy, 0.5=warning, 0=unhealthy) # TYPE health_check_database gauge health_check_database 1 # HELP health_check_database_response_time_ms Health check response time in milliseconds # TYPE health_check_database_response_time_ms gauge health_check_database_response_time_ms 2.5 # HELP php_memory_usage_bytes Current PHP memory usage in bytes # TYPE php_memory_usage_bytes gauge php_memory_usage_bytes 47185920 # HELP queue_size Total number of jobs in queue # TYPE queue_size gauge queue_size 45 # HELP queue_failed_jobs Total number of failed jobs # TYPE queue_failed_jobs counter queue_failed_jobs 2 ``` ### 3. Prometheus Configuration **prometheus.yml**: ```yaml global: scrape_interval: 15s scrape_configs: - job_name: 'app' static_configs: - targets: ['app:80'] metrics_path: '/metrics' ``` ### 4. Grafana Dashboard Import provided Grafana dashboard: `docs/deployment/grafana-dashboard.json` **Key Metrics**: - Health check status per component - Response times (p50, p95, p99) - Memory usage and trends - Queue depth and throughput - Error rates - SSL certificate expiration countdown ### 5. Alerting **Example Alertmanager Rules**: ```yaml groups: - name: application rules: - alert: HealthCheckFailed expr: health_check_database == 0 for: 5m labels: severity: critical annotations: summary: "Health check failed: {{ $labels.check }}" - alert: HighMemoryUsage expr: php_memory_usage_bytes / php_memory_limit_bytes > 0.9 for: 10m labels: severity: warning annotations: summary: "PHP memory usage above 90%" - alert: QueueBacklog expr: queue_size > 1000 for: 15m labels: severity: warning annotations: summary: "Queue backlog exceeding 1000 jobs" - alert: SSLCertificateExpiring expr: ssl_certificate_days_until_expiry < 7 for: 1h labels: severity: critical annotations: summary: "SSL certificate expiring in {{ $value }} days" ``` --- ## Logging Configuration See [production-logging.md](production-logging.md) for comprehensive logging documentation. ### 1. Production Logging Setup The framework uses `ProductionLogConfig` with optimized handlers: ```php // Configured in application bootstrap use App\Framework\Logging\ProductionLogConfig; $logConfig = ProductionLogConfig::productionWithAggregation( logPath: '/var/log/app', requestIdGenerator: $container->get(RequestIdGenerator::class) ); ``` **Features**: - Automatic log rotation (14-day retention) - Buffered writes for performance - Resilient logging with fallback - Structured JSON logs - Request/trace context - Performance metrics - Log aggregation (70-90% volume reduction) ### 2. Log Levels ``` Production: INFO and above Staging: DEBUG and above Debug Mode: ALL levels (temporary only) ``` ### 3. Log Locations ``` Primary: /var/log/app/app.log Fallback: /var/log/app/fallback.log Rotation: /var/log/app/app.log.1.gz (compressed archives) ``` ### 4. Log Monitoring ```bash # Real-time log viewing docker exec app-php tail -f /var/log/app/app.log # Search logs docker exec app-php grep "ERROR" /var/log/app/app.log | tail -20 # Check log disk usage docker exec app-php du -sh /var/log/app # View log health curl http://localhost/health/detailed | jq '.checks.logging' ``` ### 5. Log Aggregation (Optional) **ELK Stack Integration**: ```yaml # docker-compose.production.yml filebeat: image: docker.elastic.co/beats/filebeat:8.0.0 volumes: - /var/log/app:/var/log/app:ro - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro environment: - ELASTICSEARCH_HOSTS=http://elasticsearch:9200 ``` --- ## Deployment Automation ### 1. Automated Deployment Script ```bash # Deploy to production ./scripts/deployment/deploy-production.sh # With custom branch ./scripts/deployment/deploy-production.sh --branch production-v2.1.0 # Dry run (no actual changes) ./scripts/deployment/deploy-production.sh --dry-run ``` ### 2. Blue-Green Deployment ```bash # Zero-downtime deployment with blue-green strategy ./scripts/deployment/blue-green-deploy.sh # Rollback to previous version ./scripts/deployment/blue-green-rollback.sh ``` ### 3. Deployment Workflow The deployment script performs these steps: 1. **Pre-deployment**: - Validate environment configuration - Check system requirements - Backup database - Health check on current deployment 2. **Deployment**: - Pull latest code - Build Docker images - Run database migrations - Update environment files - Restart services 3. **Post-deployment**: - Health checks - Smoke tests - Log verification - Monitoring validation 4. **Cleanup**: - Remove old Docker images - Clean temporary files - Update deployment logs ### 4. CI/CD Integration **GitHub Actions Example**: ```yaml name: Deploy to Production on: push: branches: [production] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Deploy to production uses: appleboy/ssh-action@master with: host: ${{ secrets.PRODUCTION_HOST }} username: ${{ secrets.PRODUCTION_USER }} key: ${{ secrets.PRODUCTION_SSH_KEY }} script: | cd /var/www/app git pull origin production ./scripts/deployment/deploy-production.sh ``` --- ## Post-Deployment ### 1. Verification Checklist - [ ] **Services Running**: All Docker containers healthy ```bash docker-compose -f docker-compose.production.yml ps ``` - [ ] **Health Checks**: All endpoints returning healthy ```bash curl http://localhost/health/detailed | jq '.overall_healthy' ``` - [ ] **Database**: Migrations applied successfully ```bash docker exec app-php php console.php db:status ``` - [ ] **SSL**: Certificate valid and auto-renewal configured ```bash curl https://yourdomain.com/health sudo certbot renew --dry-run ``` - [ ] **Vault**: Secrets accessible ```bash curl http://localhost/health/detailed | jq '.checks.Vault' ``` - [ ] **Logging**: Logs writing correctly ```bash docker exec app-php tail -20 /var/log/app/app.log ``` - [ ] **Monitoring**: Prometheus scraping metrics ```bash curl http://localhost/metrics | head -20 ``` - [ ] **Queue**: Workers processing jobs ```bash curl http://localhost/metrics | grep queue_size ``` ### 2. Smoke Tests ```bash # Run automated smoke tests ./scripts/deployment/smoke-tests.sh # Manual smoke tests curl -I https://yourdomain.com # Homepage curl https://yourdomain.com/health # Health endpoint curl https://yourdomain.com/api/status # API endpoint ``` ### 3. Performance Testing ```bash # Basic load test with Apache Bench ab -n 1000 -c 10 https://yourdomain.com/ # Load test health endpoint ab -n 5000 -c 50 https://yourdomain.com/health/summary ``` ### 4. Monitoring Setup ```bash # Add to monitoring systems # - Prometheus target # - Grafana dashboard # - Alertmanager rules # - PagerDuty/Opsgenie integration # Verify monitoring curl http://prometheus:9090/api/v1/targets | jq ``` --- ## Troubleshooting ### Issue: Services Won't Start **Diagnosis**: ```bash docker-compose -f docker-compose.production.yml ps docker-compose -f docker-compose.production.yml logs php docker-compose -f docker-compose.production.yml logs nginx ``` **Common Causes**: - Port conflicts (80/443 already in use) - Invalid environment configuration - Missing volumes or permissions - Docker resource limits **Solutions**: ```bash # Check port usage sudo netstat -tulpn | grep :80 sudo netstat -tulpn | grep :443 # Fix permissions sudo chown -R www-data:www-data /var/log/app sudo chmod 755 /var/log/app # Recreate containers docker-compose -f docker-compose.production.yml down docker-compose -f docker-compose.production.yml up -d ``` ### Issue: Health Checks Failing **Diagnosis**: ```bash curl -v http://localhost/health/detailed docker exec app-php php console.php health:check ``` **Common Causes**: - Database connection failed - Redis connection failed - Vault inaccessible - SSL certificate expired/invalid - Queue system down **Solutions**: ```bash # Check database docker exec app-db mysql -u app_user -p -e "SELECT 1" # Check Redis docker exec app-redis redis-cli ping # Check Vault ls -la /opt/vault/production.vault docker exec app-php php -r "var_dump(file_exists('/opt/vault/production.vault'));" # Check SSL openssl x509 -in /etc/ssl/app/cert.pem -noout -dates ``` ### Issue: High Memory Usage **Diagnosis**: ```bash # Check container memory docker stats # Check PHP memory curl http://localhost/metrics | grep php_memory ``` **Solutions**: ```bash # Increase PHP memory limit # In .env.production: MEMORY_LIMIT=512M # Restart PHP service docker-compose -f docker-compose.production.yml restart php # Scale down workers if needed docker-compose -f docker-compose.production.yml up -d --scale worker=2 ``` ### Issue: Database Migration Failed **Diagnosis**: ```bash docker exec app-php php console.php db:status docker exec app-php tail -100 /var/log/app/app.log | grep migration ``` **Solutions**: ```bash # Rollback failed migration docker exec app-php php console.php db:rollback 1 # Fix migration code # Edit migration file # Test locally first # Re-run migration docker exec app-php php console.php db:migrate # Emergency: Restore from backup docker exec -i app-db mysql -u app_user -p app_production < backup.sql ``` ### Issue: SSL Certificate Problems **Diagnosis**: ```bash # Check certificate validity openssl x509 -in /etc/ssl/app/cert.pem -noout -dates # Check certificate chain openssl verify -CAfile /etc/ssl/app/ca-bundle.pem /etc/ssl/app/cert.pem # Test HTTPS curl -v https://yourdomain.com/health ``` **Solutions**: ```bash # Renew Let's Encrypt certificate sudo certbot renew --force-renewal # Update certificate paths in nginx config sudo nano /etc/nginx/sites-available/app # Restart nginx docker-compose -f docker-compose.production.yml restart nginx # Check SSL health curl http://localhost/health/detailed | jq '.checks."SSL Certificate"' ``` ### Issue: Queue Backlog **Diagnosis**: ```bash # Check queue metrics curl http://localhost/metrics | grep queue # Check worker logs docker-compose -f docker-compose.production.yml logs worker ``` **Solutions**: ```bash # Scale up workers docker-compose -f docker-compose.production.yml up -d --scale worker=5 # Check for stuck jobs docker exec app-php php console.php queue:failed # Retry failed jobs docker exec app-php php console.php queue:retry-failed # Clear queue (emergency) docker exec app-redis redis-cli FLUSHDB ``` --- ## Rollback Procedure ### 1. Quick Rollback ```bash # Use rollback script ./scripts/deployment/rollback-production.sh # Or manual rollback cd /var/www/app git checkout docker-compose -f docker-compose.production.yml up -d --build ``` ### 2. Database Rollback ```bash # Rollback migrations docker exec app-php php console.php db:rollback 3 # Restore database from backup docker exec -i app-db mysql -u app_user -p app_production < backup.sql # Verify database state docker exec app-php php console.php db:status ``` ### 3. Blue-Green Rollback ```bash # If using blue-green deployment ./scripts/deployment/blue-green-rollback.sh # This switches traffic back to previous version ``` --- ## Security Considerations ### 1. Firewall Configuration ```bash # Allow HTTP/HTTPS sudo ufw allow 80/tcp sudo ufw allow 443/tcp # Allow SSH (if needed) sudo ufw allow 22/tcp # Deny all other incoming sudo ufw default deny incoming sudo ufw default allow outgoing # Enable firewall sudo ufw enable ``` ### 2. Secrets Rotation ```bash # Rotate Vault encryption key (quarterly recommended) ./scripts/deployment/rotate-vault-key.sh # Rotate database password ./scripts/deployment/rotate-db-password.sh # Update API keys ./scripts/deployment/update-api-keys.sh ``` ### 3. Security Scanning ```bash # Scan Docker images docker scan app-php:latest # Check for vulnerabilities composer audit # Update dependencies composer update --with-all-dependencies ``` --- ## Maintenance ### 1. Regular Maintenance Tasks **Daily**: - Monitor health checks - Review error logs - Check disk space - Verify backup completion **Weekly**: - Review security alerts - Update dependencies (if needed) - Performance analysis - Log rotation verification **Monthly**: - Security patching - Certificate renewal check - Database optimization - Backup testing ### 2. Backup Strategy ```bash # Database backups (automated) 0 2 * * * /var/www/app/scripts/deployment/backup-database.sh # Volume backups (automated) 0 3 * * * /var/www/app/scripts/deployment/backup-volumes.sh # Test restore (monthly) # Restore to staging environment and verify ``` ### 3. Monitoring and Alerts Monitor these critical metrics: - Health check status (all components) - Response times (p95 < 200ms) - Error rates (< 0.1%) - Memory usage (< 80%) - Disk space (> 20% free) - SSL certificate expiration (> 30 days) - Queue depth (< 1000 jobs) --- ## Additional Resources - [Deployment Prerequisites](deployment-prerequisites.md) - [SSL Certificate Management](ssl-certificate-management.md) - [Production Environment Files](production-environment-files.md) - [Database Migration Strategy](database-migration-strategy.md) - [Vault Secrets Management](vault-secrets-management.md) - [Production Logging Configuration](production-logging.md) - [Docker Compose Production](../../docker-compose.production.yml) --- ## Support For deployment issues or questions: 1. Check [Troubleshooting](#troubleshooting) section 2. Review health checks: `curl http://localhost/health/detailed` 3. Check logs: `docker-compose logs -f` 4. Verify environment: `./scripts/deployment/validate-environment.php` For security issues, contact: security@yourdomain.com