michaelschiemer/docs/claude/deployment-architecture.md

# Deployment Architecture

Comprehensive documentation of the deployment infrastructure for the Custom PHP Framework.

## Overview

The project uses a sophisticated multi-layered deployment approach:

- **Local Development**: Docker Compose with Base+Override pattern
- **Production**: Ansible-orchestrated deployment of separate Docker stacks
- **CI/CD**: Gitea Actions triggering Ansible playbooks
- **Infrastructure**: Modular service stacks (Traefik, PostgreSQL, Registry, Gitea, Monitoring, WireGuard)

**Deployment Status**: ~95% complete (Infrastructure and Application stacks complete, CI/CD pipeline configured but not tested)

---

## Docker Compose Architecture

### Base+Override Pattern

The project uses a modern Base+Override pattern instead of a monolithic docker-compose.yml:

```
docker-compose.base.yml          # Shared service definitions
docker-compose.local.yml         # Local development overrides
docker-compose.staging.yml       # Staging environment overrides
docker-compose.production.yml    # Production environment overrides
```

**Usage**:
```bash
# Local Development
docker compose -f docker-compose.base.yml -f docker-compose.local.yml up

# Production (via Ansible)
docker compose -f docker-compose.base.yml -f docker-compose.production.yml up
```

### Legacy docker-compose.yml

**⚠️ DEPRECATED**: The root `docker-compose.yml` is marked as DEPRECATED and kept only for backward compatibility during migration.

**Planned Removal**: Q2 2025

**DO NOT USE** for new deployments - use Base+Override pattern instead.

---

## Production Deployment Configuration

### docker-compose.production.yml

Production environment configuration with security hardening:

**Key Features**:
- **Docker Secrets**: Sensitive data managed via Docker Secrets pattern
  - `DB_PASSWORD_FILE=/run/secrets/db_user_password`
  - `REDIS_PASSWORD_FILE=/run/secrets/redis_password`
  - `APP_KEY_FILE=/run/secrets/app_key`
  - `VAULT_ENCRYPTION_KEY=/run/secrets/vault_encryption_key`

- **Security Hardening**:
  - Container starts as root for gosu, drops to www-data
  - `no-new-privileges:true` security option
  - Minimal capabilities (ALL dropped, only CHOWN and DAC_OVERRIDE added)
  - Environment: `APP_ENV=production`, `APP_DEBUG=false`

- **Services**:
  - **web** (Nginx): Ports 80/443 exposed, SSL/TLS ready
  - **php** (PHP-FPM): Application runtime with security constraints
  - **redis**: Cache with Docker Secrets authentication
  - **queue-worker**: Background job processing
  - **scheduler**: Cron-like task scheduling
  - **certbot**: Automatic SSL certificate management

**Example Service Configuration**:
```yaml
php:
  restart: always
  user: "root"  # Container starts as root for gosu
  security_opt:
    - no-new-privileges:true
  cap_drop:
    - ALL
  cap_add:
    - CHOWN
    - DAC_OVERRIDE
  environment:
    - APP_ENV=production
    - APP_DEBUG=false
    - DB_PASSWORD_FILE=/run/secrets/db_user_password
  secrets:
    - db_user_password
    - redis_password
    - app_key
    - vault_encryption_key
```

---

## Deployment Directory Structure

```
deployment/
├── ansible/                    # Ansible automation
│   ├── playbooks/             # Deployment playbooks
│   │   ├── deploy-update.yml  # Application deployment/update
│   │   ├── rollback.yml       # Rollback to previous version
│   │   ├── setup-infrastructure.yml  # Infrastructure provisioning
│   │   └── system-maintenance.yml    # System maintenance tasks
│   ├── roles/                 # Ansible roles
│   │   ├── common/            # Common server setup
│   │   ├── docker/            # Docker installation
│   │   ├── firewall/          # Firewall configuration
│   │   ├── monitoring/        # Monitoring setup
│   │   ├── postgresql/        # PostgreSQL stack
│   │   ├── registry/          # Docker Registry
│   │   ├── traefik/           # Traefik reverse proxy
│   │   └── wireguard/         # WireGuard VPN
│   ├── secrets/               # Ansible Vault encrypted secrets
│   ├── templates/             # Configuration templates
│   ├── group_vars/            # Group-specific variables
│   │   └── production.yml     # Production environment config
│   ├── host_vars/             # Host-specific variables
│   ├── inventory/             # Server inventory
│   │   └── production         # Production servers
│   └── ansible.cfg            # Ansible configuration
│
├── stacks/                    # Docker Stack definitions
│   ├── traefik/               # Reverse proxy & SSL
│   │   ├── docker-compose.yml
│   │   ├── traefik.yml        # Traefik configuration
│   │   └── README.md
│   ├── postgresql/            # Database stack
│   │   ├── docker-compose.yml
│   │   └── README.md
│   ├── registry/              # Docker Registry
│   │   ├── docker-compose.yml
│   │   └── README.md
│   ├── gitea/                 # Git server & CI/CD
│   │   ├── docker-compose.yml
│   │   └── README.md
│   ├── monitoring/            # Prometheus & Grafana
│   │   ├── docker-compose.yml
│   │   ├── prometheus.yml
│   │   └── README.md
│   ├── wireguard/             # VPN access
│   │   ├── docker-compose.yml
│   │   └── README.md
│   └── application/           # Main application
│       ├── docker-compose.yml
│       ├── README.md
│       └── configs/
│
├── docs/                      # Documentation
│   ├── guides/                # How-to guides
│   │   ├── setup-guide.md     # Complete setup walkthrough
│   │   └── ...
│   ├── status/                # Deployment status
│   │   ├── deployment-summary.md
│   │   └── ...
│   ├── tests/                 # Test documentation
│   ├── history/               # Historical decisions
│   └── reference/             # Reference material
│
└── .gitea/                    # CI/CD workflows
    └── workflows/
        └── deploy.yml         # Gitea Actions deployment
```

---

## Ansible Automation

### Playbook Overview

**Infrastructure Setup** (`setup-infrastructure.yml`):
- Server provisioning and hardening
- Docker installation and configuration
- Firewall and security setup
- Stack deployment (Traefik, PostgreSQL, Registry, Gitea, Monitoring)

**Application Deployment** (`deploy-update.yml`):
- Pull latest code from Git
- Build Docker images
- Push to private registry
- Deploy application stack via docker-compose
- Run database migrations
- Restart services with zero-downtime

**Rollback** (`rollback.yml`):
- Rollback to previous Docker image tag
- Restore database from backup (if needed)
- Restart services

**System Maintenance** (`system-maintenance.yml`):
- System updates and security patches
- Docker cleanup (unused images, volumes)
- Log rotation
- Backup tasks

### Ansible Vault Secrets

**Encrypted Secrets** in `deployment/ansible/secrets/`:
- `production.vault.yml`: Production credentials (DB passwords, API keys, etc.)
- `registry.vault.yml`: Docker Registry authentication
- `gitea.vault.yml`: Gitea runner tokens and SSH keys

**Vault Operations**:
```bash
# Encrypt new file
ansible-vault encrypt deployment/ansible/secrets/production.vault.yml

# Edit encrypted file
ansible-vault edit deployment/ansible/secrets/production.vault.yml

# View encrypted file
ansible-vault view deployment/ansible/secrets/production.vault.yml

# Run playbook with vault
ansible-playbook -i inventory/production playbooks/deploy-update.yml --ask-vault-pass
```

### Inventory Configuration

**Production Inventory** (`inventory/production`):
```ini
[web_servers]
app.michaelschiemer.com ansible_host=YOUR_SERVER_IP ansible_user=deploy

[db_servers]
app.michaelschiemer.com

[all:vars]
ansible_python_interpreter=/usr/bin/python3
```

**Group Variables** (`group_vars/production.yml`):
- Domain configuration
- SSL/TLS settings
- Docker Registry URL
- Application-specific variables
- Resource limits (CPU, Memory)

---

## Docker Stacks

### Stack Architecture

Each service runs as an independent Docker stack for modularity and isolation:

#### 1. Traefik Stack
**Purpose**: Reverse proxy, SSL/TLS termination, Let's Encrypt integration

**Features**:
- Automatic SSL certificate management
- HTTP to HTTPS redirection
- Load balancing
- Service discovery via Docker labels

**Configuration**: `deployment/stacks/traefik/traefik.yml`

#### 2. PostgreSQL Stack
**Purpose**: Primary database for application

**Features**:
- Automated backups
- Connection pooling
- Replication support (optional)
- Monitoring integration

**Configuration**: `deployment/stacks/postgresql/docker-compose.yml`

#### 3. Docker Registry Stack
**Purpose**: Private Docker image registry

**Features**:
- Authentication via Basic Auth or LDAP
- SSL/TLS encryption
- Image scanning (optional)
- Integration with CI/CD pipeline

**Configuration**: `deployment/stacks/registry/docker-compose.yml`

#### 4. Gitea Stack
**Purpose**: Git server and CI/CD platform

**Features**:
- Git repository hosting
- Gitea Actions (GitHub Actions compatible)
- Gitea Runner for CI/CD execution
- Webhook support

**Configuration**: `deployment/stacks/gitea/docker-compose.yml`

#### 5. Monitoring Stack
**Purpose**: Metrics collection and visualization

**Features**:
- Prometheus for metrics collection
- Grafana for visualization
- Node Exporter for server metrics
- cAdvisor for container metrics
- Pre-configured dashboards

**Configuration**: `deployment/stacks/monitoring/docker-compose.yml`

#### 6. WireGuard Stack
**Purpose**: Secure VPN access to infrastructure

**Features**:
- Encrypted VPN tunnel
- Access to internal services
- Multi-device support

**Configuration**: `deployment/stacks/wireguard/docker-compose.yml`

#### 7. Application Stack
**Purpose**: Main PHP application

**Services**:
- **Nginx**: Web server
- **PHP-FPM**: PHP application runtime
- **Redis**: Cache and session storage
- **Queue Worker**: Background job processing
- **Scheduler**: Cron-like task scheduling

**Configuration**: `deployment/stacks/application/docker-compose.yml`

**Prerequisites**:
- Traefik stack running (for routing)
- PostgreSQL stack running (database)
- Docker Registry access (for image pulls)
- DNS configured (A records pointing to server)

---

## CI/CD Pipeline

### Gitea Actions Workflow

**Workflow File**: `.gitea/workflows/deploy.yml`

**Trigger Events**:
- Push to `main` branch (production deployment)
- Push to `staging` branch (staging deployment)
- Manual trigger via Gitea UI

**Workflow Steps**:
1. **Checkout Code**: Clone repository
2. **Build Docker Image**: Build application Docker image
3. **Push to Registry**: Push image to private Docker Registry
4. **Run Tests**: Execute test suite (PHPUnit/Pest)
5. **Deploy via Ansible**: Trigger Ansible playbook for deployment
6. **Health Check**: Verify deployment success
7. **Rollback on Failure**: Automatic rollback if health check fails

**Example Workflow**:
```yaml
name: Deploy to Production
on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Build Docker image
        run: |
          docker build -t registry.michaelschiemer.com/app:${{ github.sha }} .
          docker push registry.michaelschiemer.com/app:${{ github.sha }}

      - name: Deploy via Ansible
        run: |
          ansible-playbook -i deployment/ansible/inventory/production \
            deployment/ansible/playbooks/deploy-update.yml \
            --extra-vars "image_tag=${{ github.sha }}" \
            --vault-password-file=/secrets/vault-password
```

### Gitea Runner Setup

**Installation** (via Ansible):
- Gitea Runner installed on deployment server
- Configured with runner token from Gitea
- Systemd service for automatic start
- Docker-in-Docker (DinD) support for builds

**Configuration**:
```bash
# Register runner
gitea-runner register \
  --instance https://gitea.michaelschiemer.com \
  --token $RUNNER_TOKEN \
  --name production-runner

# Start runner
systemctl start gitea-runner
systemctl enable gitea-runner
```

---

## Deployment Workflow

### Phase 1: Initial Setup (One-Time)

**1.1 Gitea Runner Registration**:
```bash
# On deployment server
ssh deploy@app.michaelschiemer.com

# Register Gitea Runner
gitea-runner register \
  --instance https://gitea.michaelschiemer.com \
  --token <RUNNER_TOKEN> \
  --name production-runner

# Verify registration
gitea-runner list
```

**1.2 Ansible Vault Setup**:
```bash
# Create vault password file
echo "your-vault-password" > ~/.ansible/vault-password
chmod 600 ~/.ansible/vault-password

# Encrypt production secrets
cd deployment/ansible/secrets
ansible-vault create production.vault.yml
ansible-vault create registry.vault.yml
ansible-vault create gitea.vault.yml
```

**1.3 Server Provisioning**:
```bash
# Run infrastructure setup playbook
cd deployment/ansible
ansible-playbook -i inventory/production \
  playbooks/setup-infrastructure.yml \
  --vault-password-file ~/.ansible/vault-password
```

This provisions:
- Docker installation
- Firewall configuration
- All infrastructure stacks (Traefik, PostgreSQL, Registry, Gitea, Monitoring, WireGuard)

### Phase 2: Application Deployment (Repeatable)

**2.1 Manual Deployment** (via Ansible):
```bash
# Deploy/Update application
cd deployment/ansible
ansible-playbook -i inventory/production \
  playbooks/deploy-update.yml \
  --vault-password-file ~/.ansible/vault-password \
  --extra-vars "image_tag=latest"
```

**2.2 Automated Deployment** (via CI/CD):
```bash
# Push to main branch triggers automatic deployment
git push origin main

# Or trigger manually via Gitea UI
# Repository → Actions → Deploy to Production → Run Workflow
```

### Phase 3: Rollback (If Needed)

```bash
# Rollback to previous version
cd deployment/ansible
ansible-playbook -i inventory/production \
  playbooks/rollback.yml \
  --vault-password-file ~/.ansible/vault-password \
  --extra-vars "rollback_version=v1.2.3"
```

---

## Monitoring and Health Checks

### Application Health Endpoint

**URL**: `https://app.michaelschiemer.com/health`

**Response** (Healthy):
```json
{
  "status": "healthy",
  "checks": {
    "database": "ok",
    "redis": "ok",
    "filesystem": "ok"
  },
  "timestamp": "2025-01-28T12:34:56Z"
}
```

### Prometheus Metrics

**URL**: `https://metrics.michaelschiemer.com` (via WireGuard VPN)

**Key Metrics**:
- Application response times
- Database query performance
- Redis cache hit rate
- Queue job processing rate
- System resources (CPU, Memory, Disk)

### Grafana Dashboards

**URL**: `https://grafana.michaelschiemer.com` (via WireGuard VPN)

**Pre-configured Dashboards**:
- Application Overview
- Database Performance
- Queue System Metrics
- Infrastructure Health
- Docker Container Stats

---

## Troubleshooting

### Deployment Failed

**Check Ansible Logs**:
```bash
# View last deployment log
tail -f /var/log/ansible/deploy-update.log

# Check playbook output
ansible-playbook -i inventory/production \
  playbooks/deploy-update.yml \
  -vvv  # Verbose output
```

**Common Issues**:
- **Docker Registry Authentication Failed**: Check registry credentials in `secrets/registry.vault.yml`
- **Database Migration Failed**: Check database connectivity, review migration logs
- **Image Pull Failed**: Verify Docker Registry is accessible, image exists with specified tag

### Application Not Starting

**Check Docker Logs**:
```bash
# Application stack logs
ssh deploy@app.michaelschiemer.com
docker compose -f /opt/stacks/application/docker-compose.yml logs -f

# Specific service logs
docker compose logs -f php
docker compose logs -f nginx
```

**Common Issues**:
- **PHP Fatal Error**: Check PHP logs in `/var/log/app/php-error.log`
- **Database Connection Refused**: Verify PostgreSQL stack is running, check credentials
- **Redis Connection Failed**: Verify Redis stack is running, check authentication

### SSL Certificate Issues

**Check Traefik Logs**:
```bash
docker compose -f /opt/stacks/traefik/docker-compose.yml logs -f
```

**Verify Let's Encrypt Certificate**:
```bash
# Check certificate expiry
openssl s_client -connect app.michaelschiemer.com:443 -servername app.michaelschiemer.com \
  | openssl x509 -noout -dates

# Force certificate renewal (if needed)
docker exec traefik /usr/bin/traefik \
  --acme.email=admin@michaelschiemer.com \
  --certificatesresolvers.letsencrypt.acme.email=admin@michaelschiemer.com
```

### Rollback Procedure

**Immediate Rollback**:
```bash
# 1. Identify previous working version
docker images | grep app

# 2. Execute rollback playbook
cd deployment/ansible
ansible-playbook -i inventory/production \
  playbooks/rollback.yml \
  --vault-password-file ~/.ansible/vault-password \
  --extra-vars "rollback_version=<IMAGE_TAG>"

# 3. Verify health
curl -k https://app.michaelschiemer.com/health
```

---

## Best Practices

### Development Workflow

**1. Local Development**:
```bash
# Use local docker-compose override
docker compose -f docker-compose.base.yml -f docker-compose.local.yml up
```

**2. Test Before Push**:
```bash
# Run test suite
./vendor/bin/pest

# Code style check
composer cs

# Static analysis
composer analyze
```

**3. Feature Branch Deployment** (Optional):
```bash
# Create feature branch
git checkout -b feature/new-feature

# Deploy to staging (if configured)
git push origin feature/new-feature
# Triggers staging deployment
```

### Production Deployment

**1. Use Ansible for Deployments**:
- Never manually SSH and run docker commands
- Always use Ansible playbooks for consistency
- Ansible provides idempotency and rollback capability

**2. Monitor Deployments**:
- Watch Grafana dashboards during deployment
- Check application logs for errors
- Verify health endpoint returns 200 OK

**3. Database Migrations**:
- Always backup database before migrations
- Test migrations on staging first
- Migrations are automatically run by Ansible during deployment

**4. Zero-Downtime Deployments**:
- Ansible uses rolling updates for PHP-FPM workers
- Old containers remain until new containers are healthy
- Traefik automatically routes to healthy containers

### Security

**1. Secrets Management**:
- All secrets stored in Ansible Vault (encrypted)
- Production credentials never committed to Git
- Vault password stored securely (not in repository)

**2. Docker Secrets**:
- Sensitive environment variables use Docker Secrets (`_FILE` suffix)
- Secrets mounted at `/run/secrets/` (tmpfs, never written to disk)

**3. Network Isolation**:
- Services communicate via internal Docker networks
- Only Traefik exposes ports to public internet
- Database and Redis not publicly accessible

**4. SSL/TLS**:
- All traffic encrypted via Traefik
- Let's Encrypt automatic certificate renewal
- HTTP to HTTPS redirection enforced

---

## Future Improvements

### Planned Enhancements

**1. Blue-Green Deployments** (Q2 2025):
- Run two identical production environments
- Switch traffic between blue and green
- Instant rollback capability

**2. Database Replication** (Q3 2025):
- PostgreSQL primary-replica setup
- Read replicas for scaling
- Automatic failover

**3. Multi-Region Deployment** (Q4 2025):
- Deploy to multiple geographic regions
- DNS-based load balancing
- Regional failover

**4. Enhanced Monitoring** (Q1 2025):
- APM integration (Application Performance Monitoring)
- Distributed tracing
- Real-time alerting via PagerDuty/Slack

---

## Related Documentation

**Comprehensive Guides**:
- `deployment/docs/guides/setup-guide.md` - Complete 8-phase setup walkthrough
- `deployment/ansible/README.md` - Ansible automation details
- `deployment/stacks/application/README.md` - Application stack deep-dive

**Status & Progress**:
- `deployment/docs/status/deployment-summary.md` - Current deployment status (~95% complete)

**Framework Documentation**:
- `docs/claude/architecture.md` - Framework architecture overview
- `docs/claude/development-commands.md` - Development tools and commands
- `docs/claude/common-workflows.md` - Standard development workflows

---

## Quick Reference

### Common Commands

**Local Development**:
```bash
# Start local environment
docker compose -f docker-compose.base.yml -f docker-compose.local.yml up

# Run migrations
docker exec php php console.php db:migrate

# Run tests
./vendor/bin/pest
```

**Deployment (via Ansible)**:
```bash
# Deploy to production
cd deployment/ansible
ansible-playbook -i inventory/production playbooks/deploy-update.yml --ask-vault-pass

# Rollback
ansible-playbook -i inventory/production playbooks/rollback.yml --ask-vault-pass --extra-vars "rollback_version=v1.2.3"

# System maintenance
ansible-playbook -i inventory/production playbooks/system-maintenance.yml --ask-vault-pass
```

**Monitoring**:
```bash
# Check application health
curl -k https://app.michaelschiemer.com/health

# View logs
ssh deploy@app.michaelschiemer.com
docker compose -f /opt/stacks/application/docker-compose.yml logs -f

# Access Grafana (via WireGuard VPN)
https://grafana.michaelschiemer.com
```

---

**Last Updated**: 2025-01-28
**Deployment Status**: 95% Complete (CI/CD pipeline configured, pending testing)
**Next Critical Step**: Test CI/CD pipeline end-to-end