feat: CI/CD pipeline setup complete - Ansible playbooks updated, secrets configured, workflow ready

2025-10-31 01:39:24 +01:00
parent 55c04e4fd0
commit e26eb2aa12
601 changed files with 44184 additions and 32477 deletions
--- a/deployment/SETUP-GUIDE.md
+++ b/deployment/SETUP-GUIDE.md
@@ -0,0 +1,707 @@
+# Production Deployment - Complete Setup Guide
+
+**Status**: 🚧 In Progress
+**Last Updated**: 2025-10-30
+**Target Server**: 94.16.110.151 (Netcup)
+
+---
+
+## Overview
+
+This guide walks through the complete setup of production deployment from scratch, covering:
+1. Gitea Runner (Development Machine)
+2. Ansible Vault Secrets
+3. Production Server Initial Setup
+4. CI/CD Pipeline Testing
+5. Monitoring & Health Checks
+
+---
+
+## Prerequisites
+
+**Development Machine:**
+- ✅ Docker & Docker Compose installed
+- ✅ Ansible installed (`pip install ansible`)
+- ✅ SSH key for production server
+- ✅ Access to Gitea admin panel
+
+**Production Server (94.16.110.151):**
+- ✅ Docker & Docker Compose installed
+- ✅ User `deploy` created with Docker permissions
+- ✅ SSH access configured
+- ✅ Firewall configured (ports 80, 443, 2222)
+
+---
+
+## Phase 1: Gitea Runner Setup (Development Machine)
+
+### Step 1.1: Get Gitea Registration Token
+
+1. Navigate to Gitea admin panel:
+   ```
+   https://git.michaelschiemer.de/admin/actions/runners
+   ```
+
+2. Click **"Create New Runner"**
+
+3. Copy the registration token (format: `<long-random-string>`)
+
+### Step 1.2: Configure Runner Environment
+
+```bash
+cd deployment/gitea-runner
+
+# Copy environment template
+cp .env.example .env
+
+# Edit configuration
+nano .env
+```
+
+**Required Configuration in `.env`:**
+```bash
+# Gitea Instance URL
+GITEA_INSTANCE_URL=https://git.michaelschiemer.de
+
+# Registration Token (from Step 1.1)
+GITEA_RUNNER_REGISTRATION_TOKEN=<your-token-from-gitea>
+
+# Runner Name (appears in Gitea UI)
+GITEA_RUNNER_NAME=dev-runner-01
+
+# Runner Labels (environments this runner supports)
+GITEA_RUNNER_LABELS=ubuntu-latest:docker://node:20-bullseye,ubuntu-22.04:docker://catthehacker/ubuntu:act-22.04
+
+# Runner Capacity (concurrent jobs)
+GITEA_RUNNER_CAPACITY=1
+
+# Docker-in-Docker settings
+DOCKER_HOST=tcp://docker-dind:2376
+DOCKER_TLS_VERIFY=1
+```
+
+### Step 1.3: Register and Start Runner
+
+```bash
+# Register runner with Gitea
+./register.sh
+
+# Expected output:
+# ✅ Starting Gitea Runner services...
+# ✅ Runner registered successfully
+# ✅ Runner is now active
+
+# Verify runner is running
+docker compose ps
+
+# Check logs
+docker compose logs -f gitea-runner
+```
+
+### Step 1.4: Verify Runner in Gitea
+
+1. Go to: https://git.michaelschiemer.de/admin/actions/runners
+2. You should see `dev-runner-01` listed as **"Idle"** or **"Active"**
+3. Status should be green/online
+
+**✅ Checkpoint**: Runner visible in Gitea UI and showing as "Idle"
+
+---
+
+## Phase 2: Ansible Vault Secrets Setup
+
+### Step 2.1: Create Vault Password
+
+```bash
+cd deployment/ansible/secrets
+
+# Create vault password file (gitignored)
+echo "your-secure-vault-password-here" > .vault_pass
+
+# Secure the file
+chmod 600 .vault_pass
+```
+
+**⚠️ IMPORTANT**: Store this vault password in your password manager! You'll need it for all Ansible operations.
+
+### Step 2.2: Create Production Secrets File
+
+```bash
+# Copy example template
+cp production.vault.yml.example production.vault.yml
+
+# Edit with your actual secrets
+nano production.vault.yml
+```
+
+**Required Secrets in `production.vault.yml`:**
+```yaml
+---
+# Docker Registry Credentials
+docker_registry_user: "admin"
+docker_registry_password: "your-registry-password"
+
+# Application Environment Variables
+app_key: "base64:generated-32-character-key"
+app_env: "production"
+app_debug: "false"
+
+# Database Credentials
+db_host: "postgres"
+db_port: "5432"
+db_name: "framework_production"
+db_user: "framework_user"
+db_password: "your-secure-db-password"
+
+# Redis Configuration
+redis_host: "redis"
+redis_port: "6379"
+redis_password: "your-secure-redis-password"
+
+# Cache Configuration
+cache_driver: "redis"
+cache_prefix: "framework"
+
+# Queue Configuration
+queue_connection: "redis"
+queue_name: "default"
+
+# Session Configuration
+session_driver: "redis"
+session_lifetime: "120"
+
+# Encryption Keys
+encryption_key: "base64:your-32-byte-encryption-key"
+state_encryption_key: "base64:your-32-byte-state-encryption-key"
+
+# SMTP Configuration (Optional)
+mail_mailer: "smtp"
+mail_host: "smtp.example.com"
+mail_port: "587"
+mail_username: "noreply@michaelschiemer.de"
+mail_password: "your-smtp-password"
+mail_encryption: "tls"
+mail_from_address: "noreply@michaelschiemer.de"
+mail_from_name: "Framework"
+
+# Admin IPs (comma-separated)
+admin_allowed_ips: "127.0.0.1,::1"
+
+# Rate Limiting
+rate_limit_enabled: "true"
+rate_limit_default: "60"
+rate_limit_window: "60"
+```
+
+### Step 2.3: Generate Encryption Keys
+
+```bash
+# Generate app_key (32 bytes base64)
+php -r "echo 'base64:' . base64_encode(random_bytes(32)) . PHP_EOL;"
+
+# Generate encryption_key (32 bytes base64)
+php -r "echo 'base64:' . base64_encode(random_bytes(32)) . PHP_EOL;"
+
+# Generate state_encryption_key (32 bytes base64)
+php -r "echo 'base64:' . base64_encode(random_bytes(32)) . PHP_EOL;"
+
+# Copy these values into production.vault.yml
+```
+
+### Step 2.4: Encrypt Secrets File
+
+```bash
+# Encrypt the secrets file
+ansible-vault encrypt production.vault.yml \
+  --vault-password-file .vault_pass
+
+# Verify encryption worked
+file production.vault.yml
+# Should output: production.vault.yml: ASCII text
+
+# View encrypted content (should show encrypted data)
+cat production.vault.yml
+
+# Test decryption (view content)
+ansible-vault view production.vault.yml \
+  --vault-password-file .vault_pass
+```
+
+**✅ Checkpoint**: `production.vault.yml` is encrypted and can be decrypted with vault password
+
+---
+
+## Phase 3: Production Server Initial Setup
+
+### Step 3.1: Deploy Infrastructure Stacks
+
+**On Production Server (SSH as deploy user):**
+
+```bash
+# SSH to production server
+ssh deploy@94.16.110.151
+
+# Navigate to stacks directory
+cd ~/deployment/stacks
+
+# Deploy stacks in order
+
+# 1. Traefik (Reverse Proxy & SSL)
+cd traefik
+docker compose up -d
+docker compose logs -f
+# Wait for "Configuration loaded" message
+# Ctrl+C to exit logs
+
+# 2. PostgreSQL (Database)
+cd ../postgresql
+docker compose up -d
+docker compose logs -f
+# Wait for "database system is ready to accept connections"
+# Ctrl+C to exit logs
+
+# 3. Docker Registry (Private Registry)
+cd ../registry
+docker compose up -d
+docker compose logs -f
+# Wait for "listening on [::]:5000"
+# Ctrl+C to exit logs
+
+# 4. Gitea (Git Server + MySQL + Redis)
+cd ../gitea
+docker compose up -d
+docker compose logs -f
+# Wait for "Listen: http://0.0.0.0:3000"
+# Ctrl+C to exit logs
+
+# 5. Monitoring (Portainer + Grafana + Prometheus)
+cd ../monitoring
+docker compose up -d
+docker compose logs -f
+# Wait for all services to start
+# Ctrl+C to exit logs
+
+# Verify all stacks are running
+docker ps
+```
+
+### Step 3.2: Configure Gitea
+
+1. Access Gitea: https://git.michaelschiemer.de
+2. Complete initial setup wizard:
+   - Database: Use MySQL from stack
+   - Admin account: Create admin user
+   - Repository root: `/data/git/repositories`
+   - Enable Actions in admin settings
+
+### Step 3.3: Create Docker Registry User
+
+```bash
+# SSH to production server
+ssh deploy@94.16.110.151
+
+# Create registry htpasswd entry
+cd ~/deployment/stacks/registry
+docker compose exec registry htpasswd -Bbn admin your-registry-password >> auth/htpasswd
+
+# Test login
+docker login git.michaelschiemer.de:5000
+# Username: admin
+# Password: your-registry-password
+```
+
+### Step 3.4: Setup SSH Keys for Ansible
+
+**On Development Machine:**
+
+```bash
+# Generate SSH key if not exists
+ssh-keygen -t ed25519 -f ~/.ssh/production -C "ansible-deploy"
+
+# Copy public key to production server
+ssh-copy-id -i ~/.ssh/production.pub deploy@94.16.110.151
+
+# Test SSH connection
+ssh -i ~/.ssh/production deploy@94.16.110.151 "echo 'SSH works!'"
+```
+
+**✅ Checkpoint**: All infrastructure stacks running, SSH access configured
+
+---
+
+## Phase 4: Deploy Application Secrets
+
+### Step 4.1: Deploy Secrets to Production
+
+**On Development Machine:**
+
+```bash
+cd deployment/ansible
+
+# Test Ansible connectivity
+ansible production -m ping
+
+# Deploy secrets to production server
+ansible-playbook playbooks/setup-production-secrets.yml \
+  --vault-password-file secrets/.vault_pass
+
+# Expected output:
+# PLAY [Deploy Production Secrets] ***
+# TASK [Ensure secrets directory exists] *** ok
+# TASK [Deploy environment file] *** changed
+# PLAY RECAP *** production: ok=2 changed=1
+```
+
+### Step 4.2: Verify Secrets Deployed
+
+```bash
+# SSH to production server
+ssh deploy@94.16.110.151
+
+# Check secrets directory
+ls -la ~/secrets/
+
+# Verify .env.production exists (do NOT cat - contains secrets!)
+file ~/secrets/.env.production
+# Should output: .env.production: ASCII text
+
+# Check file permissions
+stat ~/secrets/.env.production
+# Should be 600 (readable only by deploy user)
+```
+
+**✅ Checkpoint**: Secrets deployed to production server in ~/secrets/.env.production
+
+---
+
+## Phase 5: Setup Gitea Secrets for CI/CD
+
+### Step 5.1: Configure Repository Secrets
+
+1. Go to repository settings in Gitea:
+   ```
+   https://git.michaelschiemer.de/<username>/michaelschiemer/settings/secrets
+   ```
+
+2. Add the following secrets:
+
+   **REGISTRY_USER**
+   ```
+   admin
+   ```
+
+   **REGISTRY_PASSWORD**
+   ```
+   <your-registry-password>
+   ```
+
+   **SSH_PRIVATE_KEY**
+   ```
+   <content-of-~/.ssh/production>
+   ```
+
+   **ANSIBLE_VAULT_PASSWORD**
+   ```
+   <your-vault-password-from-step-2.1>
+   ```
+
+### Step 5.2: Verify Secrets in Gitea
+
+1. Check secrets are visible in repository settings
+2. Each secret should show "Hidden" value with green checkmark
+
+**✅ Checkpoint**: All required secrets configured in Gitea repository
+
+---
+
+## Phase 6: First Deployment Test
+
+### Step 6.1: Manual Deployment Dry-Run
+
+**On Development Machine:**
+
+```bash
+cd deployment/ansible
+
+# Test deployment (check mode - no changes)
+ansible-playbook -i inventory/production.yml \
+  playbooks/deploy-update.yml \
+  -e "image_tag=test-$(date +%s)" \
+  -e "git_commit_sha=test123" \
+  -e "deployment_timestamp=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
+  -e "docker_registry_username=admin" \
+  -e "docker_registry_password=your-registry-password" \
+  --check
+
+# Expected: Should show what would be changed
+```
+
+### Step 6.2: Trigger CI/CD Pipeline
+
+**Option A: Push to main branch**
+```bash
+# Make a small change (add comment to file)
+echo "# Deployment test $(date)" >> deployment/DEPLOYMENT_TEST.txt
+
+# Commit and push to main
+git add deployment/DEPLOYMENT_TEST.txt
+git commit -m "test(deployment): trigger CI/CD pipeline"
+git push origin main
+```
+
+**Option B: Manual trigger**
+1. Go to Gitea repository: Actions tab
+2. Select workflow: "Production Deployment Pipeline"
+3. Click "Run workflow"
+4. Select branch: main
+5. Click "Run"
+
+### Step 6.3: Monitor Pipeline Execution
+
+1. Go to: https://git.michaelschiemer.de/<username>/michaelschiemer/actions
+2. Find the running workflow
+3. Click to view details
+4. Monitor each job:
+   - ✅ Test: Tests & quality checks pass
+   - ✅ Build: Docker image built and pushed
+   - ✅ Deploy: Application deployed to production
+
+### Step 6.4: Verify Deployment
+
+```bash
+# Test health endpoint
+curl -k https://michaelschiemer.de/health
+
+# Expected response:
+# {"status":"healthy","timestamp":"2025-10-30T14:30:00Z"}
+
+# Check application logs
+ssh deploy@94.16.110.151 "docker compose -f ~/application/docker-compose.yml logs -f app-php"
+```
+
+**✅ Checkpoint**: CI/CD pipeline executed successfully, application running on production
+
+---
+
+## Phase 7: Monitoring & Health Checks
+
+### Step 7.1: Access Monitoring Tools
+
+**Portainer**
+```
+https://portainer.michaelschiemer.de
+```
+- View all running containers
+- Monitor resource usage
+- Check logs
+
+**Grafana**
+```
+https://grafana.michaelschiemer.de
+```
+- Username: admin
+- Password: (set during setup)
+- View application metrics
+- Setup alerts
+
+**Prometheus**
+```
+https://prometheus.michaelschiemer.de
+```
+- Query metrics
+- Check targets
+- Verify scraping
+
+### Step 7.2: Configure Alerting
+
+**In Grafana:**
+
+1. Go to Alerting > Contact points
+2. Add email notification channel
+3. Create alert rules:
+   - High CPU usage (>80% for 5 minutes)
+   - High memory usage (>80%)
+   - Application down (health check fails)
+   - Database connection failures
+
+### Step 7.3: Setup Health Check Monitoring
+
+```bash
+# Create cron job on production server
+ssh deploy@94.16.110.151
+
+# Add health check script
+crontab -e
+
+# Add line:
+*/5 * * * * curl -f https://michaelschiemer.de/health || echo "Health check failed" | mail -s "Production Health Check Failed" admin@michaelschiemer.de
+```
+
+**✅ Checkpoint**: Monitoring tools accessible, alerts configured
+
+---
+
+## Phase 8: Backup & Rollback Testing
+
+### Step 8.1: Verify Backups
+
+```bash
+# SSH to production server
+ssh deploy@94.16.110.151
+
+# Check backup directory
+ls -lh ~/backups/
+
+# Should see backup folders with timestamps
+# Example: 2025-10-30T14-30-00/
+```
+
+### Step 8.2: Test Rollback
+
+```bash
+# On development machine
+cd deployment/ansible
+
+# Rollback to previous version
+ansible-playbook -i inventory/production.yml \
+  playbooks/rollback.yml
+
+# Verify rollback worked
+curl -k https://michaelschiemer.de/health
+```
+
+**✅ Checkpoint**: Backups created, rollback mechanism tested
+
+---
+
+## Verification Checklist
+
+### Infrastructure
+- [ ] Traefik running and routing HTTPS
+- [ ] PostgreSQL accessible and accepting connections
+- [ ] Docker Registry accessible at git.michaelschiemer.de:5000
+- [ ] Gitea accessible at git.michaelschiemer.de
+- [ ] Monitoring stack (Portainer, Grafana, Prometheus) running
+
+### Deployment
+- [ ] Gitea Runner registered and showing "Idle" in UI
+- [ ] Ansible Vault secrets encrypted and deployable
+- [ ] SSH access configured for Ansible
+- [ ] Repository secrets configured in Gitea
+- [ ] CI/CD pipeline runs successfully end-to-end
+
+### Application
+- [ ] Application accessible at https://michaelschiemer.de
+- [ ] Health endpoint returns 200 OK
+- [ ] Database migrations ran successfully
+- [ ] Queue workers processing jobs
+- [ ] Logs showing no errors
+
+### Monitoring
+- [ ] Portainer shows all containers running
+- [ ] Grafana dashboards displaying metrics
+- [ ] Prometheus scraping all targets
+- [ ] Alerts configured and sending notifications
+
+### Security
+- [ ] All secrets encrypted with Ansible Vault
+- [ ] SSH keys secured (600 permissions)
+- [ ] Registry requires authentication
+- [ ] HTTPS enforced on all public endpoints
+- [ ] Firewall configured correctly
+
+---
+
+## Troubleshooting
+
+### Gitea Runner Not Registering
+
+**Symptoms**: Runner not appearing in Gitea UI after running `./register.sh`
+
+**Solutions**:
+```bash
+# Check runner logs
+docker compose logs gitea-runner
+
+# Verify registration token is correct
+nano .env
+# Check GITEA_RUNNER_REGISTRATION_TOKEN
+
+# Unregister and re-register
+./unregister.sh
+./register.sh
+```
+
+### Ansible Connection Failed
+
+**Symptoms**: `Failed to connect to the host via ssh`
+
+**Solutions**:
+```bash
+# Test SSH manually
+ssh -i ~/.ssh/production deploy@94.16.110.151
+
+# Check SSH key permissions
+chmod 600 ~/.ssh/production
+
+# Verify SSH key is added to server
+ssh-copy-id -i ~/.ssh/production.pub deploy@94.16.110.151
+```
+
+### Docker Registry Authentication Failed
+
+**Symptoms**: `unauthorized: authentication required`
+
+**Solutions**:
+```bash
+# Verify credentials
+docker login git.michaelschiemer.de:5000
+# Username: admin
+# Password: <your-registry-password>
+
+# Check htpasswd file on server
+ssh deploy@94.16.110.151 "cat ~/deployment/stacks/registry/auth/htpasswd"
+```
+
+### Deployment Health Check Failed
+
+**Symptoms**: Health check returns 404 or times out
+
+**Solutions**:
+```bash
+# Check application logs
+ssh deploy@94.16.110.151 "docker compose -f ~/application/docker-compose.yml logs app-php"
+
+# Verify application stack is running
+ssh deploy@94.16.110.151 "docker ps"
+
+# Check Traefik routing
+ssh deploy@94.16.110.151 "docker compose -f ~/deployment/stacks/traefik/docker-compose.yml logs"
+```
+
+---
+
+## Next Steps
+
+After successful deployment:
+
+1. **Configure DNS**: Point michaelschiemer.de to 94.16.110.151
+2. **SSL Certificates**: Traefik will automatically request Let's Encrypt certificates
+3. **Monitoring**: Review Grafana dashboards and setup additional alerts
+4. **Backups**: Configure automated database backups
+5. **Performance**: Review application performance and optimize
+6. **Documentation**: Update team documentation with production procedures
+
+---
+
+## Support Contacts
+
+- **Infrastructure Issues**: Check Portainer logs
+- **Deployment Issues**: Review Gitea Actions logs
+- **Application Issues**: Check application logs in Portainer
+- **Emergency Rollback**: Run `ansible-playbook playbooks/rollback.yml`
+
+---
+
+**Setup Status**: 🚧 In Progress
+**Next Action**: Start with Phase 1 - Gitea Runner Setup