michaelschiemer/docs/claude/deployment-architecture.md

Deployment Architecture

Comprehensive documentation of the deployment infrastructure for the Custom PHP Framework.

Overview

The project uses a sophisticated multi-layered deployment approach:

• Local Development: Docker Compose with Base+Override pattern
• Production: Ansible-orchestrated deployment of separate Docker stacks
• CI/CD: Gitea Actions triggering Ansible playbooks
• Infrastructure: Modular service stacks (Traefik, PostgreSQL, Registry, Gitea, Monitoring, WireGuard)

Deployment Status: ~95% complete (Infrastructure and Application stacks complete, CI/CD pipeline configured but not tested)

---

Docker Compose Architecture

Base+Override Pattern

The project uses a modern Base+Override pattern instead of a monolithic docker-compose.yml:

docker-compose.base.yml          # Shared service definitions
docker-compose.local.yml         # Local development overrides
docker-compose.staging.yml       # Staging environment overrides
docker-compose.production.yml    # Production environment overrides

Usage:

# Local Development
docker compose -f docker-compose.base.yml -f docker-compose.local.yml up

# Production (via Ansible)
docker compose -f docker-compose.base.yml -f docker-compose.production.yml up

Legacy docker-compose.yml

⚠️ DEPRECATED: The root docker-compose.yml is marked as DEPRECATED and kept only for backward compatibility during migration.

Planned Removal: Q2 2025

DO NOT USE for new deployments - use Base+Override pattern instead.

---

Production Deployment Configuration

docker-compose.production.yml

Production environment configuration with security hardening:

Key Features:

• Docker Secrets: Sensitive data managed via Docker Secrets pattern

  • DB_PASSWORD_FILE=/run/secrets/db_user_password
  • REDIS_PASSWORD_FILE=/run/secrets/redis_password
  • APP_KEY_FILE=/run/secrets/app_key
  • VAULT_ENCRYPTION_KEY=/run/secrets/vault_encryption_key

• Security Hardening:

  • Container starts as root for gosu, drops to www-data
  • no-new-privileges:true security option
  • Minimal capabilities (ALL dropped, only CHOWN and DAC_OVERRIDE added)
  • Environment: APP_ENV=production, APP_DEBUG=false

• Services:

  • web (Nginx): Ports 80/443 exposed, SSL/TLS ready
  • php (PHP-FPM): Application runtime with security constraints
  • redis: Cache with Docker Secrets authentication
  • queue-worker: Background job processing
  • scheduler: Cron-like task scheduling
  • certbot: Automatic SSL certificate management

Example Service Configuration:

php:
  restart: always
  user: "root"  # Container starts as root for gosu
  security_opt:
    - no-new-privileges:true
  cap_drop:
    - ALL
  cap_add:
    - CHOWN
    - DAC_OVERRIDE
  environment:
    - APP_ENV=production
    - APP_DEBUG=false
    - DB_PASSWORD_FILE=/run/secrets/db_user_password
  secrets:
    - db_user_password
    - redis_password
    - app_key
    - vault_encryption_key

---

Deployment Directory Structure

deployment/
├── ansible/                    # Ansible automation
│   ├── playbooks/             # Deployment playbooks
│   │   ├── deploy-update.yml  # Application deployment/update
│   │   ├── rollback.yml       # Rollback to previous version
│   │   ├── setup-infrastructure.yml  # Infrastructure provisioning
│   │   └── system-maintenance.yml    # System maintenance tasks
│   ├── roles/                 # Ansible roles
│   │   ├── common/            # Common server setup
│   │   ├── docker/            # Docker installation
│   │   ├── firewall/          # Firewall configuration
│   │   ├── monitoring/        # Monitoring setup
│   │   ├── postgresql/        # PostgreSQL stack
│   │   ├── registry/          # Docker Registry
│   │   ├── traefik/           # Traefik reverse proxy
│   │   └── wireguard/         # WireGuard VPN
│   ├── secrets/               # Ansible Vault encrypted secrets
│   ├── templates/             # Configuration templates
│   ├── group_vars/            # Group-specific variables
│   │   └── production.yml     # Production environment config
│   ├── host_vars/             # Host-specific variables
│   ├── inventory/             # Server inventory
│   │   └── production         # Production servers
│   └── ansible.cfg            # Ansible configuration
│
├── stacks/                    # Docker Stack definitions
│   ├── traefik/               # Reverse proxy & SSL
│   │   ├── docker-compose.yml
│   │   ├── traefik.yml        # Traefik configuration
│   │   └── README.md
│   ├── postgresql/            # Database stack
│   │   ├── docker-compose.yml
│   │   └── README.md
│   ├── registry/              # Docker Registry
│   │   ├── docker-compose.yml
│   │   └── README.md
│   ├── gitea/                 # Git server & CI/CD
│   │   ├── docker-compose.yml
│   │   └── README.md
│   ├── monitoring/            # Prometheus & Grafana
│   │   ├── docker-compose.yml
│   │   ├── prometheus.yml
│   │   └── README.md
│   ├── wireguard/             # VPN access
│   │   ├── docker-compose.yml
│   │   └── README.md
│   └── application/           # Main application
│       ├── docker-compose.yml
│       ├── README.md
│       └── configs/
│
├── docs/                      # Documentation
│   ├── guides/                # How-to guides
│   │   ├── setup-guide.md     # Complete setup walkthrough
│   │   └── ...
│   ├── status/                # Deployment status
│   │   ├── deployment-summary.md
│   │   └── ...
│   ├── tests/                 # Test documentation
│   ├── history/               # Historical decisions
│   └── reference/             # Reference material
│
└── .gitea/                    # CI/CD workflows
    └── workflows/
        └── deploy.yml         # Gitea Actions deployment

---

Ansible Automation

Playbook Overview

Infrastructure Setup (setup-infrastructure.yml):

• Server provisioning and hardening
• Docker installation and configuration
• Firewall and security setup
• Stack deployment (Traefik, PostgreSQL, Registry, Gitea, Monitoring)

Application Deployment (deploy-update.yml):

• Pull latest code from Git
• Build Docker images
• Push to private registry
• Deploy application stack via docker-compose
• Run database migrations
• Restart services with zero-downtime

Rollback (rollback.yml):

• Rollback to previous Docker image tag
• Restore database from backup (if needed)
• Restart services

System Maintenance (system-maintenance.yml):

• System updates and security patches
• Docker cleanup (unused images, volumes)
• Log rotation
• Backup tasks

Ansible Vault Secrets

Encrypted Secrets in deployment/ansible/secrets/:

• production.vault.yml: Production credentials (DB passwords, API keys, etc.)
• registry.vault.yml: Docker Registry authentication
• gitea.vault.yml: Gitea runner tokens and SSH keys

Vault Operations:

# Encrypt new file
ansible-vault encrypt deployment/ansible/secrets/production.vault.yml

# Edit encrypted file
ansible-vault edit deployment/ansible/secrets/production.vault.yml

# View encrypted file
ansible-vault view deployment/ansible/secrets/production.vault.yml

# Run playbook with vault
ansible-playbook -i inventory/production playbooks/deploy-update.yml --ask-vault-pass

Inventory Configuration

Production Inventory (inventory/production):

[web_servers]
app.michaelschiemer.com ansible_host=YOUR_SERVER_IP ansible_user=deploy

[db_servers]
app.michaelschiemer.com

[all:vars]
ansible_python_interpreter=/usr/bin/python3

Group Variables (group_vars/production.yml):

• Domain configuration
• SSL/TLS settings
• Docker Registry URL
• Application-specific variables
• Resource limits (CPU, Memory)

---

Docker Stacks

Stack Architecture

Each service runs as an independent Docker stack for modularity and isolation:

1. Traefik Stack

Purpose: Reverse proxy, SSL/TLS termination, Let's Encrypt integration

Features:

• Automatic SSL certificate management
• HTTP to HTTPS redirection
• Load balancing
• Service discovery via Docker labels

Configuration: deployment/stacks/traefik/traefik.yml

2. PostgreSQL Stack

Purpose: Primary database for application

Features:

• Automated backups
• Connection pooling
• Replication support (optional)
• Monitoring integration

Configuration: deployment/stacks/postgresql/docker-compose.yml

3. Docker Registry Stack

Purpose: Private Docker image registry

Features:

• Authentication via Basic Auth or LDAP
• SSL/TLS encryption
• Image scanning (optional)
• Integration with CI/CD pipeline

Configuration: deployment/stacks/registry/docker-compose.yml

4. Gitea Stack

Purpose: Git server and CI/CD platform

Features:

• Git repository hosting
• Gitea Actions (GitHub Actions compatible)
• Gitea Runner for CI/CD execution
• Webhook support

Configuration: deployment/stacks/gitea/docker-compose.yml

5. Monitoring Stack

Purpose: Metrics collection and visualization

Features:

• Prometheus for metrics collection
• Grafana for visualization
• Node Exporter for server metrics
• cAdvisor for container metrics
• Pre-configured dashboards

Configuration: deployment/stacks/monitoring/docker-compose.yml

6. WireGuard Stack

Purpose: Secure VPN access to infrastructure

Features:

• Encrypted VPN tunnel
• Access to internal services
• Multi-device support

Configuration: deployment/stacks/wireguard/docker-compose.yml

7. Application Stack

Purpose: Main PHP application

Services:

• Nginx: Web server
• PHP-FPM: PHP application runtime
• Redis: Cache and session storage
• Queue Worker: Background job processing
• Scheduler: Cron-like task scheduling

Configuration: deployment/stacks/application/docker-compose.yml

Prerequisites:

• Traefik stack running (for routing)
• PostgreSQL stack running (database)
• Docker Registry access (for image pulls)
• DNS configured (A records pointing to server)

---

CI/CD Pipeline

Gitea Actions Workflow

Workflow File: .gitea/workflows/deploy.yml

Trigger Events:

• Push to main branch (production deployment)
• Push to staging branch (staging deployment)
• Manual trigger via Gitea UI

Workflow Steps:

1. Checkout Code: Clone repository
2. Build Docker Image: Build application Docker image
3. Push to Registry: Push image to private Docker Registry
4. Run Tests: Execute test suite (PHPUnit/Pest)
5. Deploy via Ansible: Trigger Ansible playbook for deployment
6. Health Check: Verify deployment success
7. Rollback on Failure: Automatic rollback if health check fails

Example Workflow:

name: Deploy to Production
on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Build Docker image
        run: |
          docker build -t registry.michaelschiemer.com/app:${{ github.sha }} .
          docker push registry.michaelschiemer.com/app:${{ github.sha }}

      - name: Deploy via Ansible
        run: |
          ansible-playbook -i deployment/ansible/inventory/production \
            deployment/ansible/playbooks/deploy-update.yml \
            --extra-vars "image_tag=${{ github.sha }}" \
            --vault-password-file=/secrets/vault-password

Gitea Runner Setup

Installation (via Ansible):

• Gitea Runner installed on deployment server
• Configured with runner token from Gitea
• Systemd service for automatic start
• Docker-in-Docker (DinD) support for builds

Configuration:

# Register runner
gitea-runner register \
  --instance https://gitea.michaelschiemer.com \
  --token $RUNNER_TOKEN \
  --name production-runner

# Start runner
systemctl start gitea-runner
systemctl enable gitea-runner

---

Deployment Workflow

Phase 1: Initial Setup (One-Time)

1.1 Gitea Runner Registration:

# On deployment server
ssh deploy@app.michaelschiemer.com

# Register Gitea Runner
gitea-runner register \
  --instance https://gitea.michaelschiemer.com \
  --token <RUNNER_TOKEN> \
  --name production-runner

# Verify registration
gitea-runner list

1.2 Ansible Vault Setup:

# Create vault password file
echo "your-vault-password" > ~/.ansible/vault-password
chmod 600 ~/.ansible/vault-password

# Encrypt production secrets
cd deployment/ansible/secrets
ansible-vault create production.vault.yml
ansible-vault create registry.vault.yml
ansible-vault create gitea.vault.yml

1.3 Server Provisioning:

# Run infrastructure setup playbook
cd deployment/ansible
ansible-playbook -i inventory/production \
  playbooks/setup-infrastructure.yml \
  --vault-password-file ~/.ansible/vault-password

This provisions:

• Docker installation
• Firewall configuration
• All infrastructure stacks (Traefik, PostgreSQL, Registry, Gitea, Monitoring, WireGuard)

Phase 2: Application Deployment (Repeatable)

2.1 Manual Deployment (via Ansible):

# Deploy/Update application
cd deployment/ansible
ansible-playbook -i inventory/production \
  playbooks/deploy-update.yml \
  --vault-password-file ~/.ansible/vault-password \
  --extra-vars "image_tag=latest"

2.2 Automated Deployment (via CI/CD):

# Push to main branch triggers automatic deployment
git push origin main

# Or trigger manually via Gitea UI
# Repository → Actions → Deploy to Production → Run Workflow

Phase 3: Rollback (If Needed)

# Rollback to previous version
cd deployment/ansible
ansible-playbook -i inventory/production \
  playbooks/rollback.yml \
  --vault-password-file ~/.ansible/vault-password \
  --extra-vars "rollback_version=v1.2.3"

---

Monitoring and Health Checks

Application Health Endpoint

URL: https://app.michaelschiemer.com/health

Response (Healthy):

{
  "status": "healthy",
  "checks": {
    "database": "ok",
    "redis": "ok",
    "filesystem": "ok"
  },
  "timestamp": "2025-01-28T12:34:56Z"
}

Prometheus Metrics

URL: https://metrics.michaelschiemer.com (via WireGuard VPN)

Key Metrics:

• Application response times
• Database query performance
• Redis cache hit rate
• Queue job processing rate
• System resources (CPU, Memory, Disk)

Grafana Dashboards

URL: https://grafana.michaelschiemer.com (via WireGuard VPN)

Pre-configured Dashboards:

• Application Overview
• Database Performance
• Queue System Metrics
• Infrastructure Health
• Docker Container Stats

---

Troubleshooting

Deployment Failed

Check Ansible Logs:

# View last deployment log
tail -f /var/log/ansible/deploy-update.log

# Check playbook output
ansible-playbook -i inventory/production \
  playbooks/deploy-update.yml \
  -vvv  # Verbose output

Common Issues:

• Docker Registry Authentication Failed: Check registry credentials in secrets/registry.vault.yml
• Database Migration Failed: Check database connectivity, review migration logs
• Image Pull Failed: Verify Docker Registry is accessible, image exists with specified tag

Application Not Starting

Check Docker Logs:

# Application stack logs
ssh deploy@app.michaelschiemer.com
docker compose -f /opt/stacks/application/docker-compose.yml logs -f

# Specific service logs
docker compose logs -f php
docker compose logs -f nginx

Common Issues:

• PHP Fatal Error: Check PHP logs in /var/log/app/php-error.log
• Database Connection Refused: Verify PostgreSQL stack is running, check credentials
• Redis Connection Failed: Verify Redis stack is running, check authentication

SSL Certificate Issues

Check Traefik Logs:

docker compose -f /opt/stacks/traefik/docker-compose.yml logs -f

Verify Let's Encrypt Certificate:

# Check certificate expiry
openssl s_client -connect app.michaelschiemer.com:443 -servername app.michaelschiemer.com \
  | openssl x509 -noout -dates

# Force certificate renewal (if needed)
docker exec traefik /usr/bin/traefik \
  --acme.email=admin@michaelschiemer.com \
  --certificatesresolvers.letsencrypt.acme.email=admin@michaelschiemer.com

Rollback Procedure

Immediate Rollback:

# 1. Identify previous working version
docker images | grep app

# 2. Execute rollback playbook
cd deployment/ansible
ansible-playbook -i inventory/production \
  playbooks/rollback.yml \
  --vault-password-file ~/.ansible/vault-password \
  --extra-vars "rollback_version=<IMAGE_TAG>"

# 3. Verify health
curl -k https://app.michaelschiemer.com/health

---

Best Practices

Development Workflow

1. Local Development:

# Use local docker-compose override
docker compose -f docker-compose.base.yml -f docker-compose.local.yml up

2. Test Before Push:

# Run test suite
./vendor/bin/pest

# Code style check
composer cs

# Static analysis
composer analyze

3. Feature Branch Deployment (Optional):

# Create feature branch
git checkout -b feature/new-feature

# Deploy to staging (if configured)
git push origin feature/new-feature
# Triggers staging deployment

Production Deployment

1. Use Ansible for Deployments:

• Never manually SSH and run docker commands
• Always use Ansible playbooks for consistency
• Ansible provides idempotency and rollback capability

2. Monitor Deployments:

• Watch Grafana dashboards during deployment
• Check application logs for errors
• Verify health endpoint returns 200 OK

3. Database Migrations:

• Always backup database before migrations
• Test migrations on staging first
• Migrations are automatically run by Ansible during deployment

4. Zero-Downtime Deployments:

• Ansible uses rolling updates for PHP-FPM workers
• Old containers remain until new containers are healthy
• Traefik automatically routes to healthy containers

Security

1. Secrets Management:

• All secrets stored in Ansible Vault (encrypted)
• Production credentials never committed to Git
• Vault password stored securely (not in repository)

2. Docker Secrets:

• Sensitive environment variables use Docker Secrets (_FILE suffix)
• Secrets mounted at /run/secrets/ (tmpfs, never written to disk)

3. Network Isolation:

• Services communicate via internal Docker networks
• Only Traefik exposes ports to public internet
• Database and Redis not publicly accessible

4. SSL/TLS:

• All traffic encrypted via Traefik
• Let's Encrypt automatic certificate renewal
• HTTP to HTTPS redirection enforced

---

Future Improvements

Planned Enhancements

1. Blue-Green Deployments (Q2 2025):

• Run two identical production environments
• Switch traffic between blue and green
• Instant rollback capability

2. Database Replication (Q3 2025):

• PostgreSQL primary-replica setup
• Read replicas for scaling
• Automatic failover

3. Multi-Region Deployment (Q4 2025):

• Deploy to multiple geographic regions
• DNS-based load balancing
• Regional failover

4. Enhanced Monitoring (Q1 2025):

• APM integration (Application Performance Monitoring)
• Distributed tracing
• Real-time alerting via PagerDuty/Slack

---

Related Documentation

Comprehensive Guides:

• deployment/docs/guides/setup-guide.md - Complete 8-phase setup walkthrough
• deployment/ansible/README.md - Ansible automation details
• deployment/stacks/application/README.md - Application stack deep-dive

Status & Progress:

• deployment/docs/status/deployment-summary.md - Current deployment status (~95% complete)

Framework Documentation:

• docs/claude/architecture.md - Framework architecture overview
• docs/claude/development-commands.md - Development tools and commands
• docs/claude/common-workflows.md - Standard development workflows

---

Quick Reference

Common Commands

Local Development:

# Start local environment
docker compose -f docker-compose.base.yml -f docker-compose.local.yml up

# Run migrations
docker exec php php console.php db:migrate

# Run tests
./vendor/bin/pest

Deployment (via Ansible):

# Deploy to production
cd deployment/ansible
ansible-playbook -i inventory/production playbooks/deploy-update.yml --ask-vault-pass

# Rollback
ansible-playbook -i inventory/production playbooks/rollback.yml --ask-vault-pass --extra-vars "rollback_version=v1.2.3"

# System maintenance
ansible-playbook -i inventory/production playbooks/system-maintenance.yml --ask-vault-pass

Monitoring:

# Check application health
curl -k https://app.michaelschiemer.com/health

# View logs
ssh deploy@app.michaelschiemer.com
docker compose -f /opt/stacks/application/docker-compose.yml logs -f

# Access Grafana (via WireGuard VPN)
https://grafana.michaelschiemer.com

---

Last Updated: 2025-01-28 Deployment Status: 95% Complete (CI/CD pipeline configured, pending testing) Next Critical Step: Test CI/CD pipeline end-to-end