michaelschiemer/deployment/README-PRODUCTION.md

# Production-Ready Deployment Infrastructure

This directory contains production-ready deployment infrastructure for the Custom PHP Framework using containerized deployment with Ansible automation.

## 🚀 Key Features

- **Container-Based Deployments**: Pre-built images, no build on production servers
- **Idempotent Operations**: Repeatable deployments with consistent results
- **Zero-Downtime Deployments**: Smart container recreation with health checks
- **Rollback Support**: Quick rollback to previous versions with tag management
- **Security Hardened**: No secrets in repo, vault-encrypted sensitive data
- **Optional CDN Integration**: Flag-based CDN configuration updates
- **Comprehensive Health Checks**: Container and HTTP health validation
- **Backup Management**: Configurable backup creation and retention

## 📁 Directory Structure

```
deployment/
├── deploy-production.sh         # Production deployment script
├── rollback-production.sh       # Production rollback script
├── infrastructure/              # Ansible automation
│   ├── ansible.cfg             # Production-hardened Ansible config
│   ├── inventories/            # Environment-specific inventories
│   │   └── production/
│   │       └── hosts.yml       # Production servers and configuration
│   ├── group_vars/             # Shared variables
│   │   └── all/
│   │       ├── main.yml        # Global configuration
│   │       └── vault.yml       # Encrypted secrets
│   ├── templates/              # Environment file templates
│   │   ├── production.env.template
│   │   └── staging.env.template
│   └── playbooks/              # Ansible automation playbooks
│       ├── deploy-application.yml  # Main deployment playbook
│       ├── rollback.yml           # Rollback playbook
│       └── update-cdn.yml         # Optional CDN update
├── applications/               # Docker Compose configurations
│   ├── docker-compose.yml      # Base compose file
│   ├── docker-compose.production.yml  # Production overlay
│   └── environments/           # Environment templates (for reference)
└── .gitignore                  # Excludes sensitive files
```

## 🛠 Prerequisites

### Required Tools
- **Ansible 2.9+** with `community.docker` collection
- **Docker** on target servers
- **SSH access** to production servers with key-based authentication
- **Vault password file** for encrypted secrets

### Infrastructure Requirements
- Pre-built container images in registry
- Production server: `94.16.110.151` with `deploy` user
- Domain: `michaelschiemer.de` with SSL certificates
- SSH key: `~/.ssh/deploy_key`

## 🔧 Configuration

### 1. Vault Password File

Create vault password file (not in repo):
```bash
# Create vault password file
echo "your_vault_password" > ~/.ansible_vault_pass
chmod 600 ~/.ansible_vault_pass

# Set environment variable
export ANSIBLE_VAULT_PASSWORD_FILE=~/.ansible_vault_pass
```

### 2. SSH Key Setup

Ensure your SSH key is properly configured:
```bash
# Copy your SSH key to the expected location
cp ~/.ssh/your_production_key ~/.ssh/deploy_key
chmod 600 ~/.ssh/deploy_key

# Test connection
ssh -i ~/.ssh/deploy_key deploy@94.16.110.151
```

### 3. Ansible Collections

Install required Ansible collections:
```bash
ansible-galaxy collection install community.docker
```

## 🚀 Deployment

### Production Deployment

Deploy a specific version to production:

```bash
# Basic deployment
./deploy-production.sh 1.2.3

# Deployment with CDN update
./deploy-production.sh 1.2.3 --cdn-update

# Deployment without backup
./deploy-production.sh 1.2.3 --no-backup

# Custom backup retention
./deploy-production.sh 1.2.3 --retention-days 60

# Using environment variables
IMAGE_TAG=1.2.3 CDN_UPDATE=true ./deploy-production.sh
```

### Rollback

Rollback to a previous version:

```bash
# Rollback to previous version
./rollback-production.sh 1.2.2

# Force rollback without confirmation
./rollback-production.sh 1.2.2 --force

# Using environment variables
ROLLBACK_TAG=1.2.2 ./rollback-production.sh
```

### Manual Ansible Commands

Run specific playbooks manually:

```bash
cd infrastructure

# Deploy application
ansible-playbook -i inventories/production/hosts.yml playbooks/deploy-application.yml \
  -e IMAGE_TAG=1.2.3 -e DOMAIN_NAME=michaelschiemer.de

# Rollback application
ansible-playbook -i inventories/production/hosts.yml playbooks/rollback.yml \
  -e ROLLBACK_TAG=1.2.2

# Update CDN only
ansible-playbook -i inventories/production/hosts.yml playbooks/update-cdn.yml \
  -e CDN_UPDATE=true
```

## 📊 Environment Variables

### Deployment Variables
```bash
IMAGE_TAG=1.2.3              # Container image tag (required)
DOMAIN_NAME=michaelschiemer.de # Target domain
CDN_UPDATE=true              # Enable CDN configuration update
BACKUP_ENABLED=true          # Enable pre-deployment backup
BACKUP_RETENTION_DAYS=30     # Backup retention period
```

### Rollback Variables
```bash
ROLLBACK_TAG=1.2.2          # Target rollback version (required)
DOMAIN_NAME=michaelschiemer.de # Target domain
```

### CI/CD Variables
```bash
ANSIBLE_VAULT_PASSWORD_FILE=~/.ansible_vault_pass # Vault password file
DB_PASSWORD=secretpassword   # Database password (encrypted in vault)
REDIS_PASSWORD=redispass    # Redis password (encrypted in vault)
MAIL_PASSWORD=mailpass      # Mail service password (encrypted in vault)
```

## 🔒 Security

### Secrets Management
- All secrets stored in `group_vars/all/vault.yml` (encrypted)
- No secrets in repository or playbooks
- Vault password via file or environment variable
- SSH key-based authentication only

### Access Control
- Deploy user with limited sudo privileges
- Container security policies enforced
- Firewall configured via base-security role
- SSH hardening enabled

### Environment File Security
- Environment files rendered from templates at deploy time
- Sensitive values injected from vault
- File permissions: 0600 (owner read/write only)
- Backup of previous versions created

## 📈 Monitoring & Health Checks

### Container Health Checks
- Individual container health monitoring
- HTTP health endpoint validation
- Configurable retry count and intervals
- Automatic failure detection

### Application Monitoring
- Health endpoint: `https://michaelschiemer.de/health`
- Container status monitoring
- Log aggregation via Docker logging drivers
- Optional Prometheus metrics

### Backup Monitoring
- Configurable backup retention
- Automatic cleanup of old backups
- Backup success/failure tracking
- Emergency recovery tag storage

## 🔄 CI/CD Integration

### GitHub Actions Example

```yaml
name: Deploy to Production

on:
  push:
    tags: ['v*']

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Ansible
        run: |
          pip install ansible
          ansible-galaxy collection install community.docker
          
      - name: Deploy to Production
        run: |
          echo "${{ secrets.ANSIBLE_VAULT_PASSWORD }}" > ~/.ansible_vault_pass
          chmod 600 ~/.ansible_vault_pass
          cd deployment
          ./deploy-production.sh ${GITHUB_REF#refs/tags/v} --no-backup
        env:
          ANSIBLE_VAULT_PASSWORD_FILE: ~/.ansible_vault_pass
          ANSIBLE_HOST_KEY_CHECKING: false
```

### Jenkins Pipeline Example

```groovy
pipeline {
    agent any
    
    environment {
        ANSIBLE_VAULT_PASSWORD_FILE = credentials('ansible-vault-password')
        IMAGE_TAG = "${params.VERSION}"
    }
    
    parameters {
        string(name: 'VERSION', defaultValue: '', description: 'Version to deploy')
        booleanParam(name: 'CDN_UPDATE', defaultValue: false, description: 'Update CDN')
    }
    
    stages {
        stage('Deploy') {
            steps {
                dir('deployment') {
                    sh """
                        ./deploy-production.sh ${IMAGE_TAG} \
                        ${params.CDN_UPDATE ? '--cdn-update' : ''}
                    """
                }
            }
        }
    }
}
```

## 🚨 Troubleshooting

### Common Issues

**1. Vault Password Issues**
```bash
# Error: vault password file not found
export ANSIBLE_VAULT_PASSWORD_FILE=~/.ansible_vault_pass
```

**2. SSH Connection Issues**
```bash
# Test SSH connection
ssh -i ~/.ssh/deploy_key deploy@94.16.110.151

# Update SSH key path in inventory if needed
```

**3. Container Health Check Failures**
```bash
# Check container logs
docker logs <container_name>

# Check health endpoint manually
curl -k https://michaelschiemer.de/health
```

**4. Permission Issues**
```bash
# Fix deployment script permissions
chmod +x deploy-production.sh rollback-production.sh
```

### Emergency Procedures

**1. Complete Rollback**
```bash
# Get last successful release
cat /var/www/html/.last_successful_release

# Rollback to last known good version
./rollback-production.sh <last_good_version> --force
```

**2. Manual Container Restart**
```bash
# SSH to server and restart containers
ssh deploy@94.16.110.151
cd /var/www/html
docker-compose restart
```

**3. Emergency Recovery**
```bash
# Use emergency recovery tag if available
cat /var/www/html/.emergency_recovery_tag
./rollback-production.sh <emergency_recovery_tag> --force
```

## 📝 Best Practices

### Deployment Best Practices
1. Always use specific version tags (not `latest`)
2. Test deployments on staging first
3. Monitor application health after deployment
4. Keep backup enabled for production
5. Use CDN updates only when static assets change

### Security Best Practices
1. Rotate vault passwords regularly
2. Keep SSH keys secure and rotated
3. Use minimal privilege principles
4. Monitor access logs
5. Keep Ansible and collections updated

### Operational Best Practices
1. Document all deployment changes
2. Monitor resource usage trends
3. Plan for backup storage requirements
4. Test rollback procedures regularly
5. Keep deployment logs for auditing

## 📞 Support

For issues with deployment infrastructure:

1. Check troubleshooting section above
2. Review Ansible logs in `infrastructure/logs/`
3. Verify container health and logs
4. Test connectivity and permissions
5. Contact system administrator if needed

---

**Note**: This infrastructure follows production-ready practices with security, reliability, and operability in mind. Always test changes in staging before applying to production.