michaelschiemer/deployment/SETUP-GUIDE.md

Production Deployment - Complete Setup Guide

Status: 🚧 In Progress Last Updated: 2025-10-30 Target Server: 94.16.110.151 (Netcup)

---

Overview

This guide walks through the complete setup of production deployment from scratch, covering:

1. Gitea Runner (Development Machine)
2. Ansible Vault Secrets
3. Production Server Initial Setup
4. CI/CD Pipeline Testing
5. Monitoring & Health Checks

---

Prerequisites

Development Machine:

• ✅ Docker & Docker Compose installed
• ✅ Ansible installed (pip install ansible)
• ✅ SSH key for production server (~/.ssh/production)
• ✅ Git SSH key configured (see Phase 0)
• ✅ Access to Gitea admin panel

Production Server (94.16.110.151):

• ✅ Docker & Docker Compose installed
• ✅ User deploy created with Docker permissions
• ✅ SSH access configured
• ✅ Firewall configured (ports 80, 443, 2222)

---

Phase 0: Git Repository SSH Access Setup (Development Machine)

Step 0.1: Generate Git SSH Key

Create a separate SSH key specifically for Git operations (different from the production server SSH key):

# Generate SSH key for Git
ssh-keygen -t ed25519 -f ~/.ssh/git_michaelschiemer -C "git@michaelschiemer.de" -N ""

# Set correct permissions
chmod 600 ~/.ssh/git_michaelschiemer
chmod 644 ~/.ssh/git_michaelschiemer.pub

Step 0.2: Configure SSH Config

Add Git SSH configuration to ~/.ssh/config:

# Edit SSH config
nano ~/.ssh/config

Add the following configuration:

Host git.michaelschiemer.de
    HostName git.michaelschiemer.de
    Port 2222
    User git
    IdentityFile ~/.ssh/git_michaelschiemer
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null

Step 0.3: Add Public Key to Gitea

1. Display your public key:

   cat ~/.ssh/git_michaelschiemer.pub
   

2. Copy the output (starts with ssh-ed25519 ...)

3. In Gitea:

   • Go to Settings → SSH / GPG Keys
   • Click Add Key
   • Paste the public key
   • Click Add Key

4. Verify the connection:

   ssh -T git@git.michaelschiemer.de
   

   Expected output: Hi there! You've successfully authenticated...

Step 0.4: Update Git Remote (if needed)

If your origin remote uses HTTPS, switch it to SSH:

# Check current remote URL
git remote -v

# Update to SSH
git remote set-url origin git@git.michaelschiemer.de:michael/michaelschiemer.git

# Test push (should work without password prompt)
git push origin main

Note: This SSH key is separate from the production server SSH key (~/.ssh/production). The production key is used for Ansible/server access, while the Git key is only for repository operations.

---

Phase 1: Gitea Runner Setup (Development Machine)

Step 1.1: Get Gitea Registration Token

1. Navigate to Gitea admin panel:

   https://git.michaelschiemer.de/admin/actions/runners
   

2. Click "Create New Runner"

3. Copy the registration token (format: <long-random-string>)

Step 1.2: Configure Runner Environment

cd deployment/gitea-runner

# Copy environment template
cp .env.example .env

# Edit configuration
nano .env

Required Configuration in .env:

# Gitea Instance URL
GITEA_INSTANCE_URL=https://git.michaelschiemer.de

# Registration Token (from Step 1.1)
GITEA_RUNNER_REGISTRATION_TOKEN=<your-token-from-gitea>

# Runner Name (appears in Gitea UI)
GITEA_RUNNER_NAME=dev-runner-01

# Runner Labels (environments this runner supports)
GITEA_RUNNER_LABELS=ubuntu-latest:docker://node:20-bullseye,ubuntu-22.04:docker://catthehacker/ubuntu:act-22.04

# Runner Capacity (concurrent jobs)
GITEA_RUNNER_CAPACITY=1

# Docker-in-Docker settings
DOCKER_HOST=tcp://docker-dind:2376
DOCKER_TLS_VERIFY=1

Step 1.3: Register and Start Runner

# Register runner with Gitea
./register.sh

# Expected output:
# ✅ Starting Gitea Runner services...
# ✅ Runner registered successfully
# ✅ Runner is now active

# Verify runner is running
docker compose ps

# Check logs
docker compose logs -f gitea-runner

Step 1.4: Verify Runner in Gitea

1. Go to: https://git.michaelschiemer.de/admin/actions/runners
2. You should see dev-runner-01 listed as "Idle" or "Active"
3. Status should be green/online

✅ Checkpoint: Runner visible in Gitea UI and showing as "Idle"

---

Phase 2: Ansible Vault Secrets Setup

Step 2.1: Create Vault Password

cd deployment/ansible/secrets

# Create vault password file (gitignored)
echo "your-secure-vault-password-here" > .vault_pass

# Secure the file
chmod 600 .vault_pass

⚠️ IMPORTANT: Store this vault password in your password manager! You'll need it for all Ansible operations.

Step 2.2: Create Production Secrets File

# Copy example template
cp production.vault.yml.example production.vault.yml

# Edit with your actual secrets
nano production.vault.yml

Required Secrets in production.vault.yml:

---
# Docker Registry Credentials
docker_registry_user: "admin"
docker_registry_password: "your-registry-password"

# Application Environment Variables
app_key: "base64:generated-32-character-key"
app_env: "production"
app_debug: "false"

# Database Credentials
db_host: "postgres"
db_port: "5432"
db_name: "framework_production"
db_user: "framework_user"
db_password: "your-secure-db-password"

# Redis Configuration
redis_host: "redis"
redis_port: "6379"
redis_password: "your-secure-redis-password"

# Cache Configuration
cache_driver: "redis"
cache_prefix: "framework"

# Queue Configuration
queue_connection: "redis"
queue_name: "default"

# Session Configuration
session_driver: "redis"
session_lifetime: "120"

# Encryption Keys
encryption_key: "base64:your-32-byte-encryption-key"
state_encryption_key: "base64:your-32-byte-state-encryption-key"

# SMTP Configuration (Optional)
mail_mailer: "smtp"
mail_host: "smtp.example.com"
mail_port: "587"
mail_username: "noreply@michaelschiemer.de"
mail_password: "your-smtp-password"
mail_encryption: "tls"
mail_from_address: "noreply@michaelschiemer.de"
mail_from_name: "Framework"

# Admin IPs (comma-separated)
admin_allowed_ips: "127.0.0.1,::1"

# Rate Limiting
rate_limit_enabled: "true"
rate_limit_default: "60"
rate_limit_window: "60"

Step 2.3: Generate Encryption Keys

# Generate app_key (32 bytes base64)
php -r "echo 'base64:' . base64_encode(random_bytes(32)) . PHP_EOL;"

# Generate encryption_key (32 bytes base64)
php -r "echo 'base64:' . base64_encode(random_bytes(32)) . PHP_EOL;"

# Generate state_encryption_key (32 bytes base64)
php -r "echo 'base64:' . base64_encode(random_bytes(32)) . PHP_EOL;"

# Copy these values into production.vault.yml

Step 2.4: Encrypt Secrets File

# Encrypt the secrets file
ansible-vault encrypt production.vault.yml \
  --vault-password-file .vault_pass

# Verify encryption worked
file production.vault.yml
# Should output: production.vault.yml: ASCII text

# View encrypted content (should show encrypted data)
cat production.vault.yml

# Test decryption (view content)
ansible-vault view production.vault.yml \
  --vault-password-file .vault_pass

✅ Checkpoint: production.vault.yml is encrypted and can be decrypted with vault password

---

Phase 3: Production Server Initial Setup

Prerequisites

Before running Phase 3, ensure:

• ✅ SSH access to production server configured (~/.ssh/production)
• ✅ Repository cloned on production server at ~/deployment/stacks (or adjust stacks_base_path in playbook)
• ✅ Ansible installed on your development machine: pip install ansible
• ✅ Ansible collections installed: ansible-galaxy collection install community.docker

Step 3.1: Clone Repository on Production Server (if not already done)

On Production Server:

# SSH to production server
ssh deploy@94.16.110.151

# Clone repository (if not already present)
mkdir -p ~/deployment
cd ~/deployment
git clone git@git.michaelschiemer.de:michael/michaelschiemer.git . || git clone https://git.michaelschiemer.de/michael/michaelschiemer.git .

Step 3.2: Deploy Infrastructure Stacks with Ansible

On Development Machine:

# Navigate to Ansible directory
cd deployment/ansible

# Run infrastructure deployment playbook
ansible-playbook playbooks/setup-infrastructure.yml \
  -i inventory/production.yml

# The playbook will:
# 1. Create required Docker networks (traefik-public, app-internal)
# 2. Deploy Traefik (Reverse Proxy & SSL)
# 3. Deploy PostgreSQL (Database)
# 4. Deploy Docker Registry (Private Registry)
# 5. Deploy Gitea (Git Server + PostgreSQL)
# 6. Deploy Monitoring (Portainer + Grafana + Prometheus)
# 7. Wait for all services to be healthy
# 8. Verify accessibility

Expected output:

• ✅ All stacks deployed successfully
• ✅ All services healthy
• ✅ Gitea accessible at https://git.michaelschiemer.de

Note: If monitoring passwords need to be stored in Vault (recommended for production), add them to secrets/production.vault.yml:

• vault_grafana_admin_password
• vault_prometheus_password

Then run the playbook with vault:

ansible-playbook playbooks/setup-infrastructure.yml \
  -i inventory/production.yml \
  --vault-password-file secrets/.vault_pass

Step 3.3: Configure Gitea (Manual Step)

1. Access Gitea: https://git.michaelschiemer.de
2. Complete initial setup wizard (first-time only):
   • Database Type: PostgreSQL
   • Database Host: postgres:5432
   • Database User: gitea
   • Database Password: gitea_password (or check deployment/stacks/gitea/docker-compose.yml)
   • Database Name: gitea
   • Admin Account: Create your admin user
   • Repository Root: /data/git/repositories (default)
3. Enable Actions (required for Phase 1):
   • Go to Site Administration → Actions
   • Enable Enable Actions checkbox
   • Save settings

Step 3.4: Verify Docker Registry

The Ansible playbook automatically creates registry authentication. To retrieve credentials:

# SSH to production server
ssh deploy@94.16.110.151

# View registry htpasswd (contains username:password hash)
cat ~/deployment/stacks/registry/auth/htpasswd

# The default username is 'admin'
# Password hash can be used to login, or create new user:
cd ~/deployment/stacks/registry
docker compose exec registry htpasswd -Bbn <username> <password> >> auth/htpasswd
docker compose restart registry

# Test login
docker login registry.michaelschiemer.de
# Or if using port:
docker login registry.michaelschiemer.de

✅ Checkpoint: All infrastructure stacks running, Gitea accessible, Actions enabled

---

Phase 4: Deploy Application Secrets

Step 4.1: Deploy Secrets to Production

On Development Machine:

cd deployment/ansible

# Test Ansible connectivity
ansible production -m ping

# Deploy secrets to production server
ansible-playbook playbooks/setup-production-secrets.yml \
  --vault-password-file secrets/.vault_pass

# Expected output:
# PLAY [Deploy Production Secrets] ***
# TASK [Ensure secrets directory exists] *** ok
# TASK [Deploy environment file] *** changed
# PLAY RECAP *** production: ok=2 changed=1

Step 4.2: Verify Secrets Deployed

# SSH to production server
ssh deploy@94.16.110.151

# Check secrets directory
ls -la ~/secrets/

# Verify .env.production exists (do NOT cat - contains secrets!)
file ~/secrets/.env.production
# Should output: .env.production: ASCII text

# Check file permissions
stat ~/secrets/.env.production
# Should be 600 (readable only by deploy user)

✅ Checkpoint: Secrets deployed to production server in ~/secrets/.env.production

---

Phase 5: Setup Gitea Secrets for CI/CD

Step 5.1: Configure Repository Secrets

1. Go to repository settings in Gitea:

   https://git.michaelschiemer.de/<username>/michaelschiemer/settings/secrets
   

2. Add the following secrets:

   REGISTRY_USER

   admin
   

   REGISTRY_PASSWORD

   <your-registry-password>
   

   SSH_PRIVATE_KEY

   <content-of-~/.ssh/production>
   

   ANSIBLE_VAULT_PASSWORD

   <your-vault-password-from-step-2.1>
   

Step 5.2: Verify Secrets in Gitea

1. Check secrets are visible in repository settings
2. Each secret should show "Hidden" value with green checkmark

✅ Checkpoint: All required secrets configured in Gitea repository

---

Phase 6: First Deployment Test

Step 6.1: Manual Deployment Dry-Run

On Development Machine:

cd deployment/ansible

# Test deployment (check mode - no changes)
ansible-playbook -i inventory/production.yml \
  playbooks/deploy-update.yml \
  -e "image_tag=test-$(date +%s)" \
  -e "git_commit_sha=test123" \
  -e "deployment_timestamp=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  -e "docker_registry_username=admin" \
  -e "docker_registry_password=your-registry-password" \
  --check

# Expected: Should show what would be changed

Step 6.2: Trigger CI/CD Pipeline

Option A: Push to main branch

# Make a small change (add comment to file)
echo "# Deployment test $(date)" >> deployment/DEPLOYMENT_TEST.txt

# Commit and push to main
git add deployment/DEPLOYMENT_TEST.txt
git commit -m "test(deployment): trigger CI/CD pipeline"
git push origin main

Option B: Manual trigger

1. Go to Gitea repository: Actions tab
2. Select workflow: "Production Deployment Pipeline"
3. Click "Run workflow"
4. Select branch: main
5. Click "Run"

Step 6.3: Monitor Pipeline Execution

1. Go to: https://git.michaelschiemer.de//michaelschiemer/actions
2. Find the running workflow
3. Click to view details
4. Monitor each job:
   • ✅ Test: Tests & quality checks pass
   • ✅ Build: Docker image built and pushed
   • ✅ Deploy: Application deployed to production

Step 6.4: Verify Deployment

# Test health endpoint
curl -k https://michaelschiemer.de/health

# Expected response:
# {"status":"healthy","timestamp":"2025-10-30T14:30:00Z"}

# Check application logs
ssh deploy@94.16.110.151 "docker compose -f ~/application/docker-compose.yml logs -f app-php"

✅ Checkpoint: CI/CD pipeline executed successfully, application running on production

---

Phase 7: Monitoring & Health Checks

Step 7.1: Access Monitoring Tools

Portainer

https://portainer.michaelschiemer.de

• View all running containers
• Monitor resource usage
• Check logs

Grafana

https://grafana.michaelschiemer.de

• Username: admin
• Password: (set during setup)
• View application metrics
• Setup alerts

Prometheus

https://prometheus.michaelschiemer.de

• Query metrics
• Check targets
• Verify scraping

Step 7.2: Configure Alerting

In Grafana:

1. Go to Alerting > Contact points
2. Add email notification channel
3. Create alert rules:
   • High CPU usage (>80% for 5 minutes)
   • High memory usage (>80%)
   • Application down (health check fails)
   • Database connection failures

Step 7.3: Setup Health Check Monitoring

# Create cron job on production server
ssh deploy@94.16.110.151

# Add health check script
crontab -e

# Add line:
*/5 * * * * curl -f https://michaelschiemer.de/health || echo "Health check failed" | mail -s "Production Health Check Failed" admin@michaelschiemer.de

✅ Checkpoint: Monitoring tools accessible, alerts configured

---

Phase 8: Backup & Rollback Testing

Step 8.1: Verify Backups

# SSH to production server
ssh deploy@94.16.110.151

# Check backup directory
ls -lh ~/backups/

# Should see backup folders with timestamps
# Example: 2025-10-30T14-30-00/

Step 8.2: Test Rollback

# On development machine
cd deployment/ansible

# Rollback to previous version
ansible-playbook -i inventory/production.yml \
  playbooks/rollback.yml

# Verify rollback worked
curl -k https://michaelschiemer.de/health

✅ Checkpoint: Backups created, rollback mechanism tested

---

Verification Checklist

Infrastructure

• Traefik running and routing HTTPS
• PostgreSQL accessible and accepting connections
• Docker Registry accessible at registry.michaelschiemer.de
• Gitea accessible at git.michaelschiemer.de
• Monitoring stack (Portainer, Grafana, Prometheus) running

Deployment

• Gitea Runner registered and showing "Idle" in UI
• Ansible Vault secrets encrypted and deployable
• SSH access configured for Ansible
• Repository secrets configured in Gitea
• CI/CD pipeline runs successfully end-to-end

Application

• Application accessible at https://michaelschiemer.de
• Health endpoint returns 200 OK
• Database migrations ran successfully
• Queue workers processing jobs
• Logs showing no errors

Monitoring

• Portainer shows all containers running
• Grafana dashboards displaying metrics
• Prometheus scraping all targets
• Alerts configured and sending notifications

Security

• All secrets encrypted with Ansible Vault
• SSH keys secured (600 permissions)
• Registry requires authentication
• HTTPS enforced on all public endpoints
• Firewall configured correctly

---

Troubleshooting

Gitea Runner Not Registering

Symptoms: Runner not appearing in Gitea UI after running ./register.sh

Solutions:

# Check runner logs
docker compose logs gitea-runner

# Verify registration token is correct
nano .env
# Check GITEA_RUNNER_REGISTRATION_TOKEN

# Unregister and re-register
./unregister.sh
./register.sh

Ansible Connection Failed

Symptoms: Failed to connect to the host via ssh

Solutions:

# Test SSH manually
ssh -i ~/.ssh/production deploy@94.16.110.151

# Check SSH key permissions
chmod 600 ~/.ssh/production

# Verify SSH key is added to server
ssh-copy-id -i ~/.ssh/production.pub deploy@94.16.110.151

Docker Registry Authentication Failed

Symptoms: unauthorized: authentication required

Solutions:

# Verify credentials
docker login registry.michaelschiemer.de
# Username: admin
# Password: <your-registry-password>

# Check htpasswd file on server
ssh deploy@94.16.110.151 "cat ~/deployment/stacks/registry/auth/htpasswd"

Deployment Health Check Failed

Symptoms: Health check returns 404 or times out

Solutions:

# Check application logs
ssh deploy@94.16.110.151 "docker compose -f ~/application/docker-compose.yml logs app-php"

# Verify application stack is running
ssh deploy@94.16.110.151 "docker ps"

# Check Traefik routing
ssh deploy@94.16.110.151 "docker compose -f ~/deployment/stacks/traefik/docker-compose.yml logs"

---

Next Steps

After successful deployment:

1. Configure DNS: Point michaelschiemer.de to 94.16.110.151
2. SSL Certificates: Traefik will automatically request Let's Encrypt certificates
3. Monitoring: Review Grafana dashboards and setup additional alerts
4. Backups: Configure automated database backups
5. Performance: Review application performance and optimize
6. Documentation: Update team documentation with production procedures

---

Support Contacts

• Infrastructure Issues: Check Portainer logs
• Deployment Issues: Review Gitea Actions logs
• Application Issues: Check application logs in Portainer
• Emergency Rollback: Run ansible-playbook playbooks/rollback.yml

---

Setup Status: 🚧 In Progress Next Action: Start with Phase 1 - Gitea Runner Setup