michael/michaelschiemer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b99765320e1b9ccdc4e788ccabd70caf34cb13d5
michaelschiemer/deployment/SETUP-GUIDE.md

16 KiB
Raw Blame History

Production Deployment - Complete Setup Guide

Status: 🚧 In Progress Last Updated: 2025-10-30 Target Server: 94.16.110.151 (Netcup)

Overview

This guide walks through the complete setup of production deployment from scratch, covering:

  1. Gitea Runner (Development Machine)
  2. Ansible Vault Secrets
  3. Production Server Initial Setup
  4. CI/CD Pipeline Testing
  5. Monitoring & Health Checks

Prerequisites

Development Machine:

  • ✅ Docker & Docker Compose installed
  • ✅ Ansible installed (pip install ansible)
  • ✅ SSH key for production server
  • ✅ Access to Gitea admin panel

Production Server (94.16.110.151):

  • ✅ Docker & Docker Compose installed
  • ✅ User deploy created with Docker permissions
  • ✅ SSH access configured
  • ✅ Firewall configured (ports 80, 443, 2222)

Phase 1: Gitea Runner Setup (Development Machine)

Step 1.1: Get Gitea Registration Token

  1. Navigate to Gitea admin panel:

    https://git.michaelschiemer.de/admin/actions/runners

  2. Click "Create New Runner"

  3. Copy the registration token (format: <long-random-string>)

Step 1.2: Configure Runner Environment

cd deployment/gitea-runner

# Copy environment template
cp .env.example .env

# Edit configuration
nano .env

Required Configuration in .env:

# Gitea Instance URL
GITEA_INSTANCE_URL=https://git.michaelschiemer.de

# Registration Token (from Step 1.1)
GITEA_RUNNER_REGISTRATION_TOKEN=<your-token-from-gitea>

# Runner Name (appears in Gitea UI)
GITEA_RUNNER_NAME=dev-runner-01

# Runner Labels (environments this runner supports)
GITEA_RUNNER_LABELS=ubuntu-latest:docker://node:20-bullseye,ubuntu-22.04:docker://catthehacker/ubuntu:act-22.04

# Runner Capacity (concurrent jobs)
GITEA_RUNNER_CAPACITY=1

# Docker-in-Docker settings
DOCKER_HOST=tcp://docker-dind:2376
DOCKER_TLS_VERIFY=1

Step 1.3: Register and Start Runner

# Register runner with Gitea
./register.sh

# Expected output:
# ✅ Starting Gitea Runner services...
# ✅ Runner registered successfully
# ✅ Runner is now active

# Verify runner is running
docker compose ps

# Check logs
docker compose logs -f gitea-runner

Step 1.4: Verify Runner in Gitea

  1. Go to: https://git.michaelschiemer.de/admin/actions/runners
  2. You should see dev-runner-01 listed as "Idle" or "Active"
  3. Status should be green/online

✅ Checkpoint: Runner visible in Gitea UI and showing as "Idle"

Phase 2: Ansible Vault Secrets Setup

Step 2.1: Create Vault Password

cd deployment/ansible/secrets

# Create vault password file (gitignored)
echo "your-secure-vault-password-here" > .vault_pass

# Secure the file
chmod 600 .vault_pass

⚠️ IMPORTANT: Store this vault password in your password manager! You'll need it for all Ansible operations.

Step 2.2: Create Production Secrets File

# Copy example template
cp production.vault.yml.example production.vault.yml

# Edit with your actual secrets
nano production.vault.yml

Required Secrets in production.vault.yml:

---
# Docker Registry Credentials
docker_registry_user: "admin"
docker_registry_password: "your-registry-password"

# Application Environment Variables
app_key: "base64:generated-32-character-key"
app_env: "production"
app_debug: "false"

# Database Credentials
db_host: "postgres"
db_port: "5432"
db_name: "framework_production"
db_user: "framework_user"
db_password: "your-secure-db-password"

# Redis Configuration
redis_host: "redis"
redis_port: "6379"
redis_password: "your-secure-redis-password"

# Cache Configuration
cache_driver: "redis"
cache_prefix: "framework"

# Queue Configuration
queue_connection: "redis"
queue_name: "default"

# Session Configuration
session_driver: "redis"
session_lifetime: "120"

# Encryption Keys
encryption_key: "base64:your-32-byte-encryption-key"
state_encryption_key: "base64:your-32-byte-state-encryption-key"

# SMTP Configuration (Optional)
mail_mailer: "smtp"
mail_host: "smtp.example.com"
mail_port: "587"
mail_username: "noreply@michaelschiemer.de"
mail_password: "your-smtp-password"
mail_encryption: "tls"
mail_from_address: "noreply@michaelschiemer.de"
mail_from_name: "Framework"

# Admin IPs (comma-separated)
admin_allowed_ips: "127.0.0.1,::1"

# Rate Limiting
rate_limit_enabled: "true"
rate_limit_default: "60"
rate_limit_window: "60"

Step 2.3: Generate Encryption Keys

# Generate app_key (32 bytes base64)
php -r "echo 'base64:' . base64_encode(random_bytes(32)) . PHP_EOL;"

# Generate encryption_key (32 bytes base64)
php -r "echo 'base64:' . base64_encode(random_bytes(32)) . PHP_EOL;"

# Generate state_encryption_key (32 bytes base64)
php -r "echo 'base64:' . base64_encode(random_bytes(32)) . PHP_EOL;"

# Copy these values into production.vault.yml

Step 2.4: Encrypt Secrets File

# Encrypt the secrets file
ansible-vault encrypt production.vault.yml \
  --vault-password-file .vault_pass

# Verify encryption worked
file production.vault.yml
# Should output: production.vault.yml: ASCII text

# View encrypted content (should show encrypted data)
cat production.vault.yml

# Test decryption (view content)
ansible-vault view production.vault.yml \
  --vault-password-file .vault_pass

✅ Checkpoint: production.vault.yml is encrypted and can be decrypted with vault password

Phase 3: Production Server Initial Setup

Step 3.1: Deploy Infrastructure Stacks

On Production Server (SSH as deploy user):

# SSH to production server
ssh deploy@94.16.110.151

# Navigate to stacks directory
cd ~/deployment/stacks

# Deploy stacks in order

# 1. Traefik (Reverse Proxy & SSL)
cd traefik
docker compose up -d
docker compose logs -f
# Wait for "Configuration loaded" message
# Ctrl+C to exit logs

# 2. PostgreSQL (Database)
cd ../postgresql
docker compose up -d
docker compose logs -f
# Wait for "database system is ready to accept connections"
# Ctrl+C to exit logs

# 3. Docker Registry (Private Registry)
cd ../registry
docker compose up -d
docker compose logs -f
# Wait for "listening on [::]:5000"
# Ctrl+C to exit logs

# 4. Gitea (Git Server + MySQL + Redis)
cd ../gitea
docker compose up -d
docker compose logs -f
# Wait for "Listen: http://0.0.0.0:3000"
# Ctrl+C to exit logs

# 5. Monitoring (Portainer + Grafana + Prometheus)
cd ../monitoring
docker compose up -d
docker compose logs -f
# Wait for all services to start
# Ctrl+C to exit logs

# Verify all stacks are running
docker ps

Step 3.2: Configure Gitea

  1. Access Gitea: https://git.michaelschiemer.de
  2. Complete initial setup wizard:
    • Database: Use MySQL from stack
    • Admin account: Create admin user
    • Repository root: /data/git/repositories
    • Enable Actions in admin settings

Step 3.3: Create Docker Registry User

# SSH to production server
ssh deploy@94.16.110.151

# Create registry htpasswd entry
cd ~/deployment/stacks/registry
docker compose exec registry htpasswd -Bbn admin your-registry-password >> auth/htpasswd

# Test login
docker login git.michaelschiemer.de:5000
# Username: admin
# Password: your-registry-password

Step 3.4: Setup SSH Keys for Ansible

On Development Machine:

# Generate SSH key if not exists
ssh-keygen -t ed25519 -f ~/.ssh/production -C "ansible-deploy"

# Copy public key to production server
ssh-copy-id -i ~/.ssh/production.pub deploy@94.16.110.151

# Test SSH connection
ssh -i ~/.ssh/production deploy@94.16.110.151 "echo 'SSH works!'"

✅ Checkpoint: All infrastructure stacks running, SSH access configured

Phase 4: Deploy Application Secrets

Step 4.1: Deploy Secrets to Production

On Development Machine:

cd deployment/ansible

# Test Ansible connectivity
ansible production -m ping

# Deploy secrets to production server
ansible-playbook playbooks/setup-production-secrets.yml \
  --vault-password-file secrets/.vault_pass

# Expected output:
# PLAY [Deploy Production Secrets] ***
# TASK [Ensure secrets directory exists] *** ok
# TASK [Deploy environment file] *** changed
# PLAY RECAP *** production: ok=2 changed=1

Step 4.2: Verify Secrets Deployed

# SSH to production server
ssh deploy@94.16.110.151

# Check secrets directory
ls -la ~/secrets/

# Verify .env.production exists (do NOT cat - contains secrets!)
file ~/secrets/.env.production
# Should output: .env.production: ASCII text

# Check file permissions
stat ~/secrets/.env.production
# Should be 600 (readable only by deploy user)

✅ Checkpoint: Secrets deployed to production server in ~/secrets/.env.production

Phase 5: Setup Gitea Secrets for CI/CD

Step 5.1: Configure Repository Secrets

  1. Go to repository settings in Gitea:

    https://git.michaelschiemer.de/<username>/michaelschiemer/settings/secrets

  2. Add the following secrets:

    REGISTRY_USER

    admin

    REGISTRY_PASSWORD

    <your-registry-password>

    SSH_PRIVATE_KEY

    <content-of-~/.ssh/production>

    ANSIBLE_VAULT_PASSWORD

    <your-vault-password-from-step-2.1>

Step 5.2: Verify Secrets in Gitea

  1. Check secrets are visible in repository settings
  2. Each secret should show "Hidden" value with green checkmark

✅ Checkpoint: All required secrets configured in Gitea repository

Phase 6: First Deployment Test

Step 6.1: Manual Deployment Dry-Run

On Development Machine:

cd deployment/ansible

# Test deployment (check mode - no changes)
ansible-playbook -i inventory/production.yml \
  playbooks/deploy-update.yml \
  -e "image_tag=test-$(date +%s)" \
  -e "git_commit_sha=test123" \
  -e "deployment_timestamp=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  -e "docker_registry_username=admin" \
  -e "docker_registry_password=your-registry-password" \
  --check

# Expected: Should show what would be changed

Step 6.2: Trigger CI/CD Pipeline

Option A: Push to main branch

# Make a small change (add comment to file)
echo "# Deployment test $(date)" >> deployment/DEPLOYMENT_TEST.txt

# Commit and push to main
git add deployment/DEPLOYMENT_TEST.txt
git commit -m "test(deployment): trigger CI/CD pipeline"
git push origin main

Option B: Manual trigger

  1. Go to Gitea repository: Actions tab
  2. Select workflow: "Production Deployment Pipeline"
  3. Click "Run workflow"
  4. Select branch: main
  5. Click "Run"

Step 6.3: Monitor Pipeline Execution

  1. Go to: https://git.michaelschiemer.de//michaelschiemer/actions
  2. Find the running workflow
  3. Click to view details
  4. Monitor each job:
    • ✅ Test: Tests & quality checks pass
    • ✅ Build: Docker image built and pushed
    • ✅ Deploy: Application deployed to production

Step 6.4: Verify Deployment

# Test health endpoint
curl -k https://michaelschiemer.de/health

# Expected response:
# {"status":"healthy","timestamp":"2025-10-30T14:30:00Z"}

# Check application logs
ssh deploy@94.16.110.151 "docker compose -f ~/application/docker-compose.yml logs -f app-php"

✅ Checkpoint: CI/CD pipeline executed successfully, application running on production

Phase 7: Monitoring & Health Checks

Step 7.1: Access Monitoring Tools

Portainer

https://portainer.michaelschiemer.de
  • View all running containers
  • Monitor resource usage
  • Check logs

Grafana

https://grafana.michaelschiemer.de
  • Username: admin
  • Password: (set during setup)
  • View application metrics
  • Setup alerts

Prometheus

https://prometheus.michaelschiemer.de
  • Query metrics
  • Check targets
  • Verify scraping

Step 7.2: Configure Alerting

In Grafana:

  1. Go to Alerting > Contact points
  2. Add email notification channel
  3. Create alert rules:
    • High CPU usage (>80% for 5 minutes)
    • High memory usage (>80%)
    • Application down (health check fails)
    • Database connection failures

Step 7.3: Setup Health Check Monitoring

# Create cron job on production server
ssh deploy@94.16.110.151

# Add health check script
crontab -e

# Add line:
*/5 * * * * curl -f https://michaelschiemer.de/health || echo "Health check failed" | mail -s "Production Health Check Failed" admin@michaelschiemer.de

✅ Checkpoint: Monitoring tools accessible, alerts configured

Phase 8: Backup & Rollback Testing

Step 8.1: Verify Backups

# SSH to production server
ssh deploy@94.16.110.151

# Check backup directory
ls -lh ~/backups/

# Should see backup folders with timestamps
# Example: 2025-10-30T14-30-00/

Step 8.2: Test Rollback

# On development machine
cd deployment/ansible

# Rollback to previous version
ansible-playbook -i inventory/production.yml \
  playbooks/rollback.yml

# Verify rollback worked
curl -k https://michaelschiemer.de/health

✅ Checkpoint: Backups created, rollback mechanism tested

Verification Checklist

Infrastructure

  • Traefik running and routing HTTPS
  • PostgreSQL accessible and accepting connections
  • Docker Registry accessible at git.michaelschiemer.de:5000
  • Gitea accessible at git.michaelschiemer.de
  • Monitoring stack (Portainer, Grafana, Prometheus) running

Deployment

  • Gitea Runner registered and showing "Idle" in UI
  • Ansible Vault secrets encrypted and deployable
  • SSH access configured for Ansible
  • Repository secrets configured in Gitea
  • CI/CD pipeline runs successfully end-to-end

Application

  • Application accessible at https://michaelschiemer.de
  • Health endpoint returns 200 OK
  • Database migrations ran successfully
  • Queue workers processing jobs
  • Logs showing no errors

Monitoring

  • Portainer shows all containers running
  • Grafana dashboards displaying metrics
  • Prometheus scraping all targets
  • Alerts configured and sending notifications

Security

  • All secrets encrypted with Ansible Vault
  • SSH keys secured (600 permissions)
  • Registry requires authentication
  • HTTPS enforced on all public endpoints
  • Firewall configured correctly

Troubleshooting

Gitea Runner Not Registering

Symptoms: Runner not appearing in Gitea UI after running ./register.sh

Solutions:

# Check runner logs
docker compose logs gitea-runner

# Verify registration token is correct
nano .env
# Check GITEA_RUNNER_REGISTRATION_TOKEN

# Unregister and re-register
./unregister.sh
./register.sh

Ansible Connection Failed

Symptoms: Failed to connect to the host via ssh

Solutions:

# Test SSH manually
ssh -i ~/.ssh/production deploy@94.16.110.151

# Check SSH key permissions
chmod 600 ~/.ssh/production

# Verify SSH key is added to server
ssh-copy-id -i ~/.ssh/production.pub deploy@94.16.110.151

Docker Registry Authentication Failed

Symptoms: unauthorized: authentication required

Solutions:

# Verify credentials
docker login git.michaelschiemer.de:5000
# Username: admin
# Password: <your-registry-password>

# Check htpasswd file on server
ssh deploy@94.16.110.151 "cat ~/deployment/stacks/registry/auth/htpasswd"

Deployment Health Check Failed

Symptoms: Health check returns 404 or times out

Solutions:

# Check application logs
ssh deploy@94.16.110.151 "docker compose -f ~/application/docker-compose.yml logs app-php"

# Verify application stack is running
ssh deploy@94.16.110.151 "docker ps"

# Check Traefik routing
ssh deploy@94.16.110.151 "docker compose -f ~/deployment/stacks/traefik/docker-compose.yml logs"

Next Steps

After successful deployment:

  1. Configure DNS: Point michaelschiemer.de to 94.16.110.151
  2. SSL Certificates: Traefik will automatically request Let's Encrypt certificates
  3. Monitoring: Review Grafana dashboards and setup additional alerts
  4. Backups: Configure automated database backups
  5. Performance: Review application performance and optimize
  6. Documentation: Update team documentation with production procedures

Support Contacts

  • Infrastructure Issues: Check Portainer logs
  • Deployment Issues: Review Gitea Actions logs
  • Application Issues: Check application logs in Portainer
  • Emergency Rollback: Run ansible-playbook playbooks/rollback.yml

Setup Status: 🚧 In Progress Next Action: Start with Phase 1 - Gitea Runner Setup

Reference in New Issue View Git Blame Copy Permalink