feat: add comprehensive framework features and deployment improvements

Major additions:
- Storage abstraction layer with filesystem and in-memory implementations
- Gitea API integration with MCP tools for repository management
- Console dialog mode with interactive command execution
- WireGuard VPN DNS fix implementation and documentation
- HTTP client streaming response support
- Router generic result type
- Parameter type validator for framework core

Framework enhancements:
- Console command registry improvements
- Console dialog components
- Method signature analyzer updates
- Route mapper refinements
- MCP server and tool mapper updates
- Queue job chain and dependency commands
- Discovery tokenizer improvements

Infrastructure:
- Deployment architecture documentation
- Ansible playbook updates for WireGuard client regeneration
- Production environment configuration updates
- Docker Compose local configuration updates
- Remove obsolete docker-compose.yml (replaced by environment-specific configs)

Documentation:
- PERMISSIONS.md for access control guidelines
- WireGuard DNS fix implementation details
- Console dialog mode usage guide
- Deployment architecture overview

Testing:
- Multi-purpose attribute tests
- Gitea Actions integration tests (typed and untyped)
This commit is contained in:
2025-11-04 20:39:48 +01:00
parent 700fe8118b
commit 3ed2685e74
80 changed files with 9891 additions and 850 deletions

View File

@@ -0,0 +1,777 @@
# Deployment Architecture
Comprehensive documentation of the deployment infrastructure for the Custom PHP Framework.
## Overview
The project uses a sophisticated multi-layered deployment approach:
- **Local Development**: Docker Compose with Base+Override pattern
- **Production**: Ansible-orchestrated deployment of separate Docker stacks
- **CI/CD**: Gitea Actions triggering Ansible playbooks
- **Infrastructure**: Modular service stacks (Traefik, PostgreSQL, Registry, Gitea, Monitoring, WireGuard)
**Deployment Status**: ~95% complete (Infrastructure and Application stacks complete, CI/CD pipeline configured but not tested)
---
## Docker Compose Architecture
### Base+Override Pattern
The project uses a modern Base+Override pattern instead of a monolithic docker-compose.yml:
```
docker-compose.base.yml # Shared service definitions
docker-compose.local.yml # Local development overrides
docker-compose.staging.yml # Staging environment overrides
docker-compose.production.yml # Production environment overrides
```
**Usage**:
```bash
# Local Development
docker compose -f docker-compose.base.yml -f docker-compose.local.yml up
# Production (via Ansible)
docker compose -f docker-compose.base.yml -f docker-compose.production.yml up
```
### Legacy docker-compose.yml
**⚠️ DEPRECATED**: The root `docker-compose.yml` is marked as DEPRECATED and kept only for backward compatibility during migration.
**Planned Removal**: Q2 2025
**DO NOT USE** for new deployments - use Base+Override pattern instead.
---
## Production Deployment Configuration
### docker-compose.production.yml
Production environment configuration with security hardening:
**Key Features**:
- **Docker Secrets**: Sensitive data managed via Docker Secrets pattern
- `DB_PASSWORD_FILE=/run/secrets/db_user_password`
- `REDIS_PASSWORD_FILE=/run/secrets/redis_password`
- `APP_KEY_FILE=/run/secrets/app_key`
- `VAULT_ENCRYPTION_KEY=/run/secrets/vault_encryption_key`
- **Security Hardening**:
- Container starts as root for gosu, drops to www-data
- `no-new-privileges:true` security option
- Minimal capabilities (ALL dropped, only CHOWN and DAC_OVERRIDE added)
- Environment: `APP_ENV=production`, `APP_DEBUG=false`
- **Services**:
- **web** (Nginx): Ports 80/443 exposed, SSL/TLS ready
- **php** (PHP-FPM): Application runtime with security constraints
- **redis**: Cache with Docker Secrets authentication
- **queue-worker**: Background job processing
- **scheduler**: Cron-like task scheduling
- **certbot**: Automatic SSL certificate management
**Example Service Configuration**:
```yaml
php:
restart: always
user: "root" # Container starts as root for gosu
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
cap_add:
- CHOWN
- DAC_OVERRIDE
environment:
- APP_ENV=production
- APP_DEBUG=false
- DB_PASSWORD_FILE=/run/secrets/db_user_password
secrets:
- db_user_password
- redis_password
- app_key
- vault_encryption_key
```
---
## Deployment Directory Structure
```
deployment/
├── ansible/ # Ansible automation
│ ├── playbooks/ # Deployment playbooks
│ │ ├── deploy-update.yml # Application deployment/update
│ │ ├── rollback.yml # Rollback to previous version
│ │ ├── setup-infrastructure.yml # Infrastructure provisioning
│ │ └── system-maintenance.yml # System maintenance tasks
│ ├── roles/ # Ansible roles
│ │ ├── common/ # Common server setup
│ │ ├── docker/ # Docker installation
│ │ ├── firewall/ # Firewall configuration
│ │ ├── monitoring/ # Monitoring setup
│ │ ├── postgresql/ # PostgreSQL stack
│ │ ├── registry/ # Docker Registry
│ │ ├── traefik/ # Traefik reverse proxy
│ │ └── wireguard/ # WireGuard VPN
│ ├── secrets/ # Ansible Vault encrypted secrets
│ ├── templates/ # Configuration templates
│ ├── group_vars/ # Group-specific variables
│ │ └── production.yml # Production environment config
│ ├── host_vars/ # Host-specific variables
│ ├── inventory/ # Server inventory
│ │ └── production # Production servers
│ └── ansible.cfg # Ansible configuration
├── stacks/ # Docker Stack definitions
│ ├── traefik/ # Reverse proxy & SSL
│ │ ├── docker-compose.yml
│ │ ├── traefik.yml # Traefik configuration
│ │ └── README.md
│ ├── postgresql/ # Database stack
│ │ ├── docker-compose.yml
│ │ └── README.md
│ ├── registry/ # Docker Registry
│ │ ├── docker-compose.yml
│ │ └── README.md
│ ├── gitea/ # Git server & CI/CD
│ │ ├── docker-compose.yml
│ │ └── README.md
│ ├── monitoring/ # Prometheus & Grafana
│ │ ├── docker-compose.yml
│ │ ├── prometheus.yml
│ │ └── README.md
│ ├── wireguard/ # VPN access
│ │ ├── docker-compose.yml
│ │ └── README.md
│ └── application/ # Main application
│ ├── docker-compose.yml
│ ├── README.md
│ └── configs/
├── docs/ # Documentation
│ ├── guides/ # How-to guides
│ │ ├── setup-guide.md # Complete setup walkthrough
│ │ └── ...
│ ├── status/ # Deployment status
│ │ ├── deployment-summary.md
│ │ └── ...
│ ├── tests/ # Test documentation
│ ├── history/ # Historical decisions
│ └── reference/ # Reference material
└── .gitea/ # CI/CD workflows
└── workflows/
└── deploy.yml # Gitea Actions deployment
```
---
## Ansible Automation
### Playbook Overview
**Infrastructure Setup** (`setup-infrastructure.yml`):
- Server provisioning and hardening
- Docker installation and configuration
- Firewall and security setup
- Stack deployment (Traefik, PostgreSQL, Registry, Gitea, Monitoring)
**Application Deployment** (`deploy-update.yml`):
- Pull latest code from Git
- Build Docker images
- Push to private registry
- Deploy application stack via docker-compose
- Run database migrations
- Restart services with zero-downtime
**Rollback** (`rollback.yml`):
- Rollback to previous Docker image tag
- Restore database from backup (if needed)
- Restart services
**System Maintenance** (`system-maintenance.yml`):
- System updates and security patches
- Docker cleanup (unused images, volumes)
- Log rotation
- Backup tasks
### Ansible Vault Secrets
**Encrypted Secrets** in `deployment/ansible/secrets/`:
- `production.vault.yml`: Production credentials (DB passwords, API keys, etc.)
- `registry.vault.yml`: Docker Registry authentication
- `gitea.vault.yml`: Gitea runner tokens and SSH keys
**Vault Operations**:
```bash
# Encrypt new file
ansible-vault encrypt deployment/ansible/secrets/production.vault.yml
# Edit encrypted file
ansible-vault edit deployment/ansible/secrets/production.vault.yml
# View encrypted file
ansible-vault view deployment/ansible/secrets/production.vault.yml
# Run playbook with vault
ansible-playbook -i inventory/production playbooks/deploy-update.yml --ask-vault-pass
```
### Inventory Configuration
**Production Inventory** (`inventory/production`):
```ini
[web_servers]
app.michaelschiemer.com ansible_host=YOUR_SERVER_IP ansible_user=deploy
[db_servers]
app.michaelschiemer.com
[all:vars]
ansible_python_interpreter=/usr/bin/python3
```
**Group Variables** (`group_vars/production.yml`):
- Domain configuration
- SSL/TLS settings
- Docker Registry URL
- Application-specific variables
- Resource limits (CPU, Memory)
---
## Docker Stacks
### Stack Architecture
Each service runs as an independent Docker stack for modularity and isolation:
#### 1. Traefik Stack
**Purpose**: Reverse proxy, SSL/TLS termination, Let's Encrypt integration
**Features**:
- Automatic SSL certificate management
- HTTP to HTTPS redirection
- Load balancing
- Service discovery via Docker labels
**Configuration**: `deployment/stacks/traefik/traefik.yml`
#### 2. PostgreSQL Stack
**Purpose**: Primary database for application
**Features**:
- Automated backups
- Connection pooling
- Replication support (optional)
- Monitoring integration
**Configuration**: `deployment/stacks/postgresql/docker-compose.yml`
#### 3. Docker Registry Stack
**Purpose**: Private Docker image registry
**Features**:
- Authentication via Basic Auth or LDAP
- SSL/TLS encryption
- Image scanning (optional)
- Integration with CI/CD pipeline
**Configuration**: `deployment/stacks/registry/docker-compose.yml`
#### 4. Gitea Stack
**Purpose**: Git server and CI/CD platform
**Features**:
- Git repository hosting
- Gitea Actions (GitHub Actions compatible)
- Gitea Runner for CI/CD execution
- Webhook support
**Configuration**: `deployment/stacks/gitea/docker-compose.yml`
#### 5. Monitoring Stack
**Purpose**: Metrics collection and visualization
**Features**:
- Prometheus for metrics collection
- Grafana for visualization
- Node Exporter for server metrics
- cAdvisor for container metrics
- Pre-configured dashboards
**Configuration**: `deployment/stacks/monitoring/docker-compose.yml`
#### 6. WireGuard Stack
**Purpose**: Secure VPN access to infrastructure
**Features**:
- Encrypted VPN tunnel
- Access to internal services
- Multi-device support
**Configuration**: `deployment/stacks/wireguard/docker-compose.yml`
#### 7. Application Stack
**Purpose**: Main PHP application
**Services**:
- **Nginx**: Web server
- **PHP-FPM**: PHP application runtime
- **Redis**: Cache and session storage
- **Queue Worker**: Background job processing
- **Scheduler**: Cron-like task scheduling
**Configuration**: `deployment/stacks/application/docker-compose.yml`
**Prerequisites**:
- Traefik stack running (for routing)
- PostgreSQL stack running (database)
- Docker Registry access (for image pulls)
- DNS configured (A records pointing to server)
---
## CI/CD Pipeline
### Gitea Actions Workflow
**Workflow File**: `.gitea/workflows/deploy.yml`
**Trigger Events**:
- Push to `main` branch (production deployment)
- Push to `staging` branch (staging deployment)
- Manual trigger via Gitea UI
**Workflow Steps**:
1. **Checkout Code**: Clone repository
2. **Build Docker Image**: Build application Docker image
3. **Push to Registry**: Push image to private Docker Registry
4. **Run Tests**: Execute test suite (PHPUnit/Pest)
5. **Deploy via Ansible**: Trigger Ansible playbook for deployment
6. **Health Check**: Verify deployment success
7. **Rollback on Failure**: Automatic rollback if health check fails
**Example Workflow**:
```yaml
name: Deploy to Production
on:
push:
branches: [main]
workflow_dispatch:
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Build Docker image
run: |
docker build -t registry.michaelschiemer.com/app:${{ github.sha }} .
docker push registry.michaelschiemer.com/app:${{ github.sha }}
- name: Deploy via Ansible
run: |
ansible-playbook -i deployment/ansible/inventory/production \
deployment/ansible/playbooks/deploy-update.yml \
--extra-vars "image_tag=${{ github.sha }}" \
--vault-password-file=/secrets/vault-password
```
### Gitea Runner Setup
**Installation** (via Ansible):
- Gitea Runner installed on deployment server
- Configured with runner token from Gitea
- Systemd service for automatic start
- Docker-in-Docker (DinD) support for builds
**Configuration**:
```bash
# Register runner
gitea-runner register \
--instance https://gitea.michaelschiemer.com \
--token $RUNNER_TOKEN \
--name production-runner
# Start runner
systemctl start gitea-runner
systemctl enable gitea-runner
```
---
## Deployment Workflow
### Phase 1: Initial Setup (One-Time)
**1.1 Gitea Runner Registration**:
```bash
# On deployment server
ssh deploy@app.michaelschiemer.com
# Register Gitea Runner
gitea-runner register \
--instance https://gitea.michaelschiemer.com \
--token <RUNNER_TOKEN> \
--name production-runner
# Verify registration
gitea-runner list
```
**1.2 Ansible Vault Setup**:
```bash
# Create vault password file
echo "your-vault-password" > ~/.ansible/vault-password
chmod 600 ~/.ansible/vault-password
# Encrypt production secrets
cd deployment/ansible/secrets
ansible-vault create production.vault.yml
ansible-vault create registry.vault.yml
ansible-vault create gitea.vault.yml
```
**1.3 Server Provisioning**:
```bash
# Run infrastructure setup playbook
cd deployment/ansible
ansible-playbook -i inventory/production \
playbooks/setup-infrastructure.yml \
--vault-password-file ~/.ansible/vault-password
```
This provisions:
- Docker installation
- Firewall configuration
- All infrastructure stacks (Traefik, PostgreSQL, Registry, Gitea, Monitoring, WireGuard)
### Phase 2: Application Deployment (Repeatable)
**2.1 Manual Deployment** (via Ansible):
```bash
# Deploy/Update application
cd deployment/ansible
ansible-playbook -i inventory/production \
playbooks/deploy-update.yml \
--vault-password-file ~/.ansible/vault-password \
--extra-vars "image_tag=latest"
```
**2.2 Automated Deployment** (via CI/CD):
```bash
# Push to main branch triggers automatic deployment
git push origin main
# Or trigger manually via Gitea UI
# Repository → Actions → Deploy to Production → Run Workflow
```
### Phase 3: Rollback (If Needed)
```bash
# Rollback to previous version
cd deployment/ansible
ansible-playbook -i inventory/production \
playbooks/rollback.yml \
--vault-password-file ~/.ansible/vault-password \
--extra-vars "rollback_version=v1.2.3"
```
---
## Monitoring and Health Checks
### Application Health Endpoint
**URL**: `https://app.michaelschiemer.com/health`
**Response** (Healthy):
```json
{
"status": "healthy",
"checks": {
"database": "ok",
"redis": "ok",
"filesystem": "ok"
},
"timestamp": "2025-01-28T12:34:56Z"
}
```
### Prometheus Metrics
**URL**: `https://metrics.michaelschiemer.com` (via WireGuard VPN)
**Key Metrics**:
- Application response times
- Database query performance
- Redis cache hit rate
- Queue job processing rate
- System resources (CPU, Memory, Disk)
### Grafana Dashboards
**URL**: `https://grafana.michaelschiemer.com` (via WireGuard VPN)
**Pre-configured Dashboards**:
- Application Overview
- Database Performance
- Queue System Metrics
- Infrastructure Health
- Docker Container Stats
---
## Troubleshooting
### Deployment Failed
**Check Ansible Logs**:
```bash
# View last deployment log
tail -f /var/log/ansible/deploy-update.log
# Check playbook output
ansible-playbook -i inventory/production \
playbooks/deploy-update.yml \
-vvv # Verbose output
```
**Common Issues**:
- **Docker Registry Authentication Failed**: Check registry credentials in `secrets/registry.vault.yml`
- **Database Migration Failed**: Check database connectivity, review migration logs
- **Image Pull Failed**: Verify Docker Registry is accessible, image exists with specified tag
### Application Not Starting
**Check Docker Logs**:
```bash
# Application stack logs
ssh deploy@app.michaelschiemer.com
docker compose -f /opt/stacks/application/docker-compose.yml logs -f
# Specific service logs
docker compose logs -f php
docker compose logs -f nginx
```
**Common Issues**:
- **PHP Fatal Error**: Check PHP logs in `/var/log/app/php-error.log`
- **Database Connection Refused**: Verify PostgreSQL stack is running, check credentials
- **Redis Connection Failed**: Verify Redis stack is running, check authentication
### SSL Certificate Issues
**Check Traefik Logs**:
```bash
docker compose -f /opt/stacks/traefik/docker-compose.yml logs -f
```
**Verify Let's Encrypt Certificate**:
```bash
# Check certificate expiry
openssl s_client -connect app.michaelschiemer.com:443 -servername app.michaelschiemer.com \
| openssl x509 -noout -dates
# Force certificate renewal (if needed)
docker exec traefik /usr/bin/traefik \
--acme.email=admin@michaelschiemer.com \
--certificatesresolvers.letsencrypt.acme.email=admin@michaelschiemer.com
```
### Rollback Procedure
**Immediate Rollback**:
```bash
# 1. Identify previous working version
docker images | grep app
# 2. Execute rollback playbook
cd deployment/ansible
ansible-playbook -i inventory/production \
playbooks/rollback.yml \
--vault-password-file ~/.ansible/vault-password \
--extra-vars "rollback_version=<IMAGE_TAG>"
# 3. Verify health
curl -k https://app.michaelschiemer.com/health
```
---
## Best Practices
### Development Workflow
**1. Local Development**:
```bash
# Use local docker-compose override
docker compose -f docker-compose.base.yml -f docker-compose.local.yml up
```
**2. Test Before Push**:
```bash
# Run test suite
./vendor/bin/pest
# Code style check
composer cs
# Static analysis
composer analyze
```
**3. Feature Branch Deployment** (Optional):
```bash
# Create feature branch
git checkout -b feature/new-feature
# Deploy to staging (if configured)
git push origin feature/new-feature
# Triggers staging deployment
```
### Production Deployment
**1. Use Ansible for Deployments**:
- Never manually SSH and run docker commands
- Always use Ansible playbooks for consistency
- Ansible provides idempotency and rollback capability
**2. Monitor Deployments**:
- Watch Grafana dashboards during deployment
- Check application logs for errors
- Verify health endpoint returns 200 OK
**3. Database Migrations**:
- Always backup database before migrations
- Test migrations on staging first
- Migrations are automatically run by Ansible during deployment
**4. Zero-Downtime Deployments**:
- Ansible uses rolling updates for PHP-FPM workers
- Old containers remain until new containers are healthy
- Traefik automatically routes to healthy containers
### Security
**1. Secrets Management**:
- All secrets stored in Ansible Vault (encrypted)
- Production credentials never committed to Git
- Vault password stored securely (not in repository)
**2. Docker Secrets**:
- Sensitive environment variables use Docker Secrets (`_FILE` suffix)
- Secrets mounted at `/run/secrets/` (tmpfs, never written to disk)
**3. Network Isolation**:
- Services communicate via internal Docker networks
- Only Traefik exposes ports to public internet
- Database and Redis not publicly accessible
**4. SSL/TLS**:
- All traffic encrypted via Traefik
- Let's Encrypt automatic certificate renewal
- HTTP to HTTPS redirection enforced
---
## Future Improvements
### Planned Enhancements
**1. Blue-Green Deployments** (Q2 2025):
- Run two identical production environments
- Switch traffic between blue and green
- Instant rollback capability
**2. Database Replication** (Q3 2025):
- PostgreSQL primary-replica setup
- Read replicas for scaling
- Automatic failover
**3. Multi-Region Deployment** (Q4 2025):
- Deploy to multiple geographic regions
- DNS-based load balancing
- Regional failover
**4. Enhanced Monitoring** (Q1 2025):
- APM integration (Application Performance Monitoring)
- Distributed tracing
- Real-time alerting via PagerDuty/Slack
---
## Related Documentation
**Comprehensive Guides**:
- `deployment/docs/guides/setup-guide.md` - Complete 8-phase setup walkthrough
- `deployment/ansible/README.md` - Ansible automation details
- `deployment/stacks/application/README.md` - Application stack deep-dive
**Status & Progress**:
- `deployment/docs/status/deployment-summary.md` - Current deployment status (~95% complete)
**Framework Documentation**:
- `docs/claude/architecture.md` - Framework architecture overview
- `docs/claude/development-commands.md` - Development tools and commands
- `docs/claude/common-workflows.md` - Standard development workflows
---
## Quick Reference
### Common Commands
**Local Development**:
```bash
# Start local environment
docker compose -f docker-compose.base.yml -f docker-compose.local.yml up
# Run migrations
docker exec php php console.php db:migrate
# Run tests
./vendor/bin/pest
```
**Deployment (via Ansible)**:
```bash
# Deploy to production
cd deployment/ansible
ansible-playbook -i inventory/production playbooks/deploy-update.yml --ask-vault-pass
# Rollback
ansible-playbook -i inventory/production playbooks/rollback.yml --ask-vault-pass --extra-vars "rollback_version=v1.2.3"
# System maintenance
ansible-playbook -i inventory/production playbooks/system-maintenance.yml --ask-vault-pass
```
**Monitoring**:
```bash
# Check application health
curl -k https://app.michaelschiemer.com/health
# View logs
ssh deploy@app.michaelschiemer.com
docker compose -f /opt/stacks/application/docker-compose.yml logs -f
# Access Grafana (via WireGuard VPN)
https://grafana.michaelschiemer.com
```
---
**Last Updated**: 2025-01-28
**Deployment Status**: 95% Complete (CI/CD pipeline configured, pending testing)
**Next Critical Step**: Test CI/CD pipeline end-to-end