michaelschiemer/deployment/SETUP-GUIDE.md

# Production Deployment - Complete Setup Guide

**Status**: 🚧 In Progress
**Last Updated**: 2025-10-30
**Target Server**: 94.16.110.151 (Netcup)

---

## Overview

This guide walks through the complete setup of production deployment from scratch, covering:
1. Gitea Runner (Development Machine)
2. Ansible Vault Secrets
3. Production Server Initial Setup
4. CI/CD Pipeline Testing
5. Monitoring & Health Checks

---

## Prerequisites

**Development Machine:**
- ✅ Docker & Docker Compose installed
- ✅ Ansible installed (`pip install ansible`)
- ✅ SSH key for production server (`~/.ssh/production`)
- ✅ Git SSH key configured (see Phase 0)
- ✅ Access to Gitea admin panel

**Production Server (94.16.110.151):**
- ✅ Docker & Docker Compose installed
- ✅ User `deploy` created with Docker permissions
- ✅ SSH access configured
- ✅ Firewall configured (ports 80, 443, 2222)

---

## Phase 0: Git Repository SSH Access Setup (Development Machine)

### Step 0.1: Generate Git SSH Key

Create a separate SSH key specifically for Git operations (different from the production server SSH key):

```bash
# Generate SSH key for Git
ssh-keygen -t ed25519 -f ~/.ssh/git_michaelschiemer -C "git@michaelschiemer.de" -N ""

# Set correct permissions
chmod 600 ~/.ssh/git_michaelschiemer
chmod 644 ~/.ssh/git_michaelschiemer.pub
```

### Step 0.2: Configure SSH Config

Add Git SSH configuration to `~/.ssh/config`:

```bash
# Edit SSH config
nano ~/.ssh/config
```

Add the following configuration:

```
Host git.michaelschiemer.de
    HostName git.michaelschiemer.de
    Port 2222
    User git
    IdentityFile ~/.ssh/git_michaelschiemer
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
```

### Step 0.3: Add Public Key to Gitea

1. Display your public key:
   ```bash
   cat ~/.ssh/git_michaelschiemer.pub
   ```

2. Copy the output (starts with `ssh-ed25519 ...`)

3. In Gitea:
   - Go to **Settings** → **SSH / GPG Keys**
   - Click **Add Key**
   - Paste the public key
   - Click **Add Key**

4. Verify the connection:
   ```bash
   ssh -T git@git.michaelschiemer.de
   ```

   Expected output: `Hi there! You've successfully authenticated...`

### Step 0.4: Update Git Remote (if needed)

If your `origin` remote uses HTTPS, switch it to SSH:

```bash
# Check current remote URL
git remote -v

# Update to SSH
git remote set-url origin git@git.michaelschiemer.de:michael/michaelschiemer.git

# Test push (should work without password prompt)
git push origin main
```

**Note**: This SSH key is separate from the production server SSH key (`~/.ssh/production`). The production key is used for Ansible/server access, while the Git key is only for repository operations.

---

## Phase 1: Gitea Runner Setup (Development Machine)

### Step 1.1: Get Gitea Registration Token

1. Navigate to Gitea admin panel:
   ```
   https://git.michaelschiemer.de/admin/actions/runners
   ```

2. Click **"Create New Runner"**

3. Copy the registration token (format: `<long-random-string>`)

### Step 1.2: Configure Runner Environment

```bash
cd deployment/gitea-runner

# Copy environment template
cp .env.example .env

# Edit configuration
nano .env
```

**Required Configuration in `.env`:**
```bash
# Gitea Instance URL
GITEA_INSTANCE_URL=https://git.michaelschiemer.de

# Registration Token (from Step 1.1)
GITEA_RUNNER_REGISTRATION_TOKEN=<your-token-from-gitea>

# Runner Name (appears in Gitea UI)
GITEA_RUNNER_NAME=dev-runner-01

# Runner Labels (environments this runner supports)
GITEA_RUNNER_LABELS=ubuntu-latest:docker://node:20-bullseye,ubuntu-22.04:docker://catthehacker/ubuntu:act-22.04

# Runner Capacity (concurrent jobs)
GITEA_RUNNER_CAPACITY=1

# Docker-in-Docker settings
DOCKER_HOST=tcp://docker-dind:2376
DOCKER_TLS_VERIFY=1
```

### Step 1.3: Register and Start Runner

```bash
# Register runner with Gitea
./register.sh

# Expected output:
# ✅ Starting Gitea Runner services...
# ✅ Runner registered successfully
# ✅ Runner is now active

# Verify runner is running
docker compose ps

# Check logs
docker compose logs -f gitea-runner
```

### Step 1.4: Verify Runner in Gitea

1. Go to: https://git.michaelschiemer.de/admin/actions/runners
2. You should see `dev-runner-01` listed as **"Idle"** or **"Active"**
3. Status should be green/online

**✅ Checkpoint**: Runner visible in Gitea UI and showing as "Idle"

---

## Phase 2: Ansible Vault Secrets Setup

### Step 2.1: Create Vault Password

```bash
cd deployment/ansible/secrets

# Create vault password file (gitignored)
echo "your-secure-vault-password-here" > .vault_pass

# Secure the file
chmod 600 .vault_pass
```

**⚠️ IMPORTANT**: Store this vault password in your password manager! You'll need it for all Ansible operations.

### Step 2.2: Create Production Secrets File

```bash
# Copy example template
cp production.vault.yml.example production.vault.yml

# Edit with your actual secrets
nano production.vault.yml
```

**Required Secrets in `production.vault.yml`:**
```yaml
---
# Docker Registry Credentials
docker_registry_user: "admin"
docker_registry_password: "your-registry-password"

# Application Environment Variables
app_key: "base64:generated-32-character-key"
app_env: "production"
app_debug: "false"

# Database Credentials
db_host: "postgres"
db_port: "5432"
db_name: "framework_production"
db_user: "framework_user"
db_password: "your-secure-db-password"

# Redis Configuration
redis_host: "redis"
redis_port: "6379"
redis_password: "your-secure-redis-password"

# Cache Configuration
cache_driver: "redis"
cache_prefix: "framework"

# Queue Configuration
queue_connection: "redis"
queue_name: "default"

# Session Configuration
session_driver: "redis"
session_lifetime: "120"

# Encryption Keys
encryption_key: "base64:your-32-byte-encryption-key"
state_encryption_key: "base64:your-32-byte-state-encryption-key"

# SMTP Configuration (Optional)
mail_mailer: "smtp"
mail_host: "smtp.example.com"
mail_port: "587"
mail_username: "noreply@michaelschiemer.de"
mail_password: "your-smtp-password"
mail_encryption: "tls"
mail_from_address: "noreply@michaelschiemer.de"
mail_from_name: "Framework"

# Admin IPs (comma-separated)
admin_allowed_ips: "127.0.0.1,::1"

# Rate Limiting
rate_limit_enabled: "true"
rate_limit_default: "60"
rate_limit_window: "60"
```

### Step 2.3: Generate Encryption Keys

```bash
# Generate app_key (32 bytes base64)
php -r "echo 'base64:' . base64_encode(random_bytes(32)) . PHP_EOL;"

# Generate encryption_key (32 bytes base64)
php -r "echo 'base64:' . base64_encode(random_bytes(32)) . PHP_EOL;"

# Generate state_encryption_key (32 bytes base64)
php -r "echo 'base64:' . base64_encode(random_bytes(32)) . PHP_EOL;"

# Copy these values into production.vault.yml
```

### Step 2.4: Encrypt Secrets File

```bash
# Encrypt the secrets file
ansible-vault encrypt production.vault.yml \
  --vault-password-file .vault_pass

# Verify encryption worked
file production.vault.yml
# Should output: production.vault.yml: ASCII text

# View encrypted content (should show encrypted data)
cat production.vault.yml

# Test decryption (view content)
ansible-vault view production.vault.yml \
  --vault-password-file .vault_pass
```

**✅ Checkpoint**: `production.vault.yml` is encrypted and can be decrypted with vault password

---

## Phase 3: Production Server Initial Setup

### Prerequisites

Before running Phase 3, ensure:
- ✅ SSH access to production server configured (`~/.ssh/production`)
- ✅ Repository cloned on production server at `~/deployment/stacks` (or adjust `stacks_base_path` in playbook)
- ✅ Ansible installed on your development machine: `pip install ansible`
- ✅ Ansible collections installed: `ansible-galaxy collection install community.docker`

### Step 3.1: Clone Repository on Production Server (if not already done)

**On Production Server:**

```bash
# SSH to production server
ssh deploy@94.16.110.151

# Clone repository (if not already present)
mkdir -p ~/deployment
cd ~/deployment
git clone git@git.michaelschiemer.de:michael/michaelschiemer.git . || git clone https://git.michaelschiemer.de/michael/michaelschiemer.git .
```

### Step 3.2: Deploy Infrastructure Stacks with Ansible

**On Development Machine:**

```bash
# Navigate to Ansible directory
cd deployment/ansible

# Run infrastructure deployment playbook
ansible-playbook playbooks/setup-infrastructure.yml \
  -i inventory/production.yml

# The playbook will:
# 1. Create required Docker networks (traefik-public, app-internal)
# 2. Deploy Traefik (Reverse Proxy & SSL)
# 3. Deploy PostgreSQL (Database)
# 4. Deploy Docker Registry (Private Registry)
# 5. Deploy Gitea (Git Server + PostgreSQL)
# 6. Deploy Monitoring (Portainer + Grafana + Prometheus)
# 7. Wait for all services to be healthy
# 8. Verify accessibility
```

**Expected output:**
- ✅ All stacks deployed successfully
- ✅ All services healthy
- ✅ Gitea accessible at https://git.michaelschiemer.de

**Note:** If monitoring passwords need to be stored in Vault (recommended for production), add them to `secrets/production.vault.yml`:
- `vault_grafana_admin_password`
- `vault_prometheus_password`

Then run the playbook with vault:
```bash
ansible-playbook playbooks/setup-infrastructure.yml \
  -i inventory/production.yml \
  --vault-password-file secrets/.vault_pass
```

### Step 3.3: Configure Gitea (Manual Step)

1. Access Gitea: https://git.michaelschiemer.de
2. Complete initial setup wizard (first-time only):
   - **Database Type**: PostgreSQL
   - **Database Host**: `postgres:5432`
   - **Database User**: `gitea`
   - **Database Password**: `gitea_password` (or check `deployment/stacks/gitea/docker-compose.yml`)
   - **Database Name**: `gitea`
   - **Admin Account**: Create your admin user
   - **Repository Root**: `/data/git/repositories` (default)
3. **Enable Actions** (required for Phase 1):
   - Go to **Site Administration** → **Actions**
   - Enable **Enable Actions** checkbox
   - Save settings

### Step 3.4: Verify Docker Registry

The Ansible playbook automatically creates registry authentication. To retrieve credentials:

```bash
# SSH to production server
ssh deploy@94.16.110.151

# View registry htpasswd (contains username:password hash)
cat ~/deployment/stacks/registry/auth/htpasswd

# The default username is 'admin'
# Password hash can be used to login, or create new user:
cd ~/deployment/stacks/registry
docker compose exec registry htpasswd -Bbn <username> <password> >> auth/htpasswd
docker compose restart registry

# Test login
docker login registry.michaelschiemer.de
# Or if using port:
docker login registry.michaelschiemer.de
```

**✅ Checkpoint**: All infrastructure stacks running, Gitea accessible, Actions enabled

---

## Phase 4: Deploy Application Secrets

### Step 4.1: Deploy Secrets to Production

**On Development Machine:**

```bash
cd deployment/ansible

# Test Ansible connectivity
ansible production -m ping

# Deploy secrets to production server
ansible-playbook playbooks/setup-production-secrets.yml \
  --vault-password-file secrets/.vault_pass

# Expected output:
# PLAY [Deploy Production Secrets] ***
# TASK [Ensure secrets directory exists] *** ok
# TASK [Deploy environment file] *** changed
# PLAY RECAP *** production: ok=2 changed=1
```

### Step 4.2: Verify Secrets Deployed

```bash
# SSH to production server
ssh deploy@94.16.110.151

# Check secrets directory
ls -la ~/secrets/

# Verify .env.production exists (do NOT cat - contains secrets!)
file ~/secrets/.env.production
# Should output: .env.production: ASCII text

# Check file permissions
stat ~/secrets/.env.production
# Should be 600 (readable only by deploy user)
```

**✅ Checkpoint**: Secrets deployed to production server in ~/secrets/.env.production

---

## Phase 5: Setup Gitea Secrets for CI/CD

### Step 5.1: Configure Repository Secrets

1. Go to repository settings in Gitea:
   ```
   https://git.michaelschiemer.de/<username>/michaelschiemer/settings/secrets
   ```

2. Add the following secrets:

   **REGISTRY_USER**
   ```
   admin
   ```

   **REGISTRY_PASSWORD**
   ```
   <your-registry-password>
   ```

   **SSH_PRIVATE_KEY**
   ```
   <content-of-~/.ssh/production>
   ```

   **ANSIBLE_VAULT_PASSWORD**
   ```
   <your-vault-password-from-step-2.1>
   ```

### Step 5.2: Verify Secrets in Gitea

1. Check secrets are visible in repository settings
2. Each secret should show "Hidden" value with green checkmark

**✅ Checkpoint**: All required secrets configured in Gitea repository

---

## Phase 6: First Deployment Test

### Step 6.1: Manual Deployment Dry-Run

**On Development Machine:**

```bash
cd deployment/ansible

# Test deployment (check mode - no changes)
ansible-playbook -i inventory/production.yml \
  playbooks/deploy-update.yml \
  -e "image_tag=test-$(date +%s)" \
  -e "git_commit_sha=test123" \
  -e "deployment_timestamp=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  -e "docker_registry_username=admin" \
  -e "docker_registry_password=your-registry-password" \
  --check

# Expected: Should show what would be changed
```

### Step 6.2: Trigger CI/CD Pipeline

**Option A: Push to main branch**
```bash
# Make a small change (add comment to file)
echo "# Deployment test $(date)" >> deployment/DEPLOYMENT_TEST.txt

# Commit and push to main
git add deployment/DEPLOYMENT_TEST.txt
git commit -m "test(deployment): trigger CI/CD pipeline"
git push origin main
```

**Option B: Manual trigger**
1. Go to Gitea repository: Actions tab
2. Select workflow: "Production Deployment Pipeline"
3. Click "Run workflow"
4. Select branch: main
5. Click "Run"

### Step 6.3: Monitor Pipeline Execution

1. Go to: https://git.michaelschiemer.de/<username>/michaelschiemer/actions
2. Find the running workflow
3. Click to view details
4. Monitor each job:
   - ✅ Test: Tests & quality checks pass
   - ✅ Build: Docker image built and pushed
   - ✅ Deploy: Application deployed to production

### Step 6.4: Verify Deployment

```bash
# Test health endpoint
curl -k https://michaelschiemer.de/health

# Expected response:
# {"status":"healthy","timestamp":"2025-10-30T14:30:00Z"}

# Check application logs
ssh deploy@94.16.110.151 "docker compose -f ~/application/docker-compose.yml logs -f app-php"
```

**✅ Checkpoint**: CI/CD pipeline executed successfully, application running on production

---

## Phase 7: Monitoring & Health Checks

### Step 7.1: Access Monitoring Tools

**Portainer**
```
https://portainer.michaelschiemer.de
```
- View all running containers
- Monitor resource usage
- Check logs

**Grafana**
```
https://grafana.michaelschiemer.de
```
- Username: admin
- Password: (set during setup)
- View application metrics
- Setup alerts

**Prometheus**
```
https://prometheus.michaelschiemer.de
```
- Query metrics
- Check targets
- Verify scraping

### Step 7.2: Configure Alerting

**In Grafana:**

1. Go to Alerting > Contact points
2. Add email notification channel
3. Create alert rules:
   - High CPU usage (>80% for 5 minutes)
   - High memory usage (>80%)
   - Application down (health check fails)
   - Database connection failures

### Step 7.3: Setup Health Check Monitoring

```bash
# Create cron job on production server
ssh deploy@94.16.110.151

# Add health check script
crontab -e

# Add line:
*/5 * * * * curl -f https://michaelschiemer.de/health || echo "Health check failed" | mail -s "Production Health Check Failed" admin@michaelschiemer.de
```

**✅ Checkpoint**: Monitoring tools accessible, alerts configured

---

## Phase 8: Backup & Rollback Testing

### Step 8.1: Verify Backups

```bash
# SSH to production server
ssh deploy@94.16.110.151

# Check backup directory
ls -lh ~/backups/

# Should see backup folders with timestamps
# Example: 2025-10-30T14-30-00/
```

### Step 8.2: Test Rollback

```bash
# On development machine
cd deployment/ansible

# Rollback to previous version
ansible-playbook -i inventory/production.yml \
  playbooks/rollback.yml

# Verify rollback worked
curl -k https://michaelschiemer.de/health
```

**✅ Checkpoint**: Backups created, rollback mechanism tested

---

## Verification Checklist

### Infrastructure
- [ ] Traefik running and routing HTTPS
- [ ] PostgreSQL accessible and accepting connections
- [ ] Docker Registry accessible at registry.michaelschiemer.de
- [ ] Gitea accessible at git.michaelschiemer.de
- [ ] Monitoring stack (Portainer, Grafana, Prometheus) running

### Deployment
- [ ] Gitea Runner registered and showing "Idle" in UI
- [ ] Ansible Vault secrets encrypted and deployable
- [ ] SSH access configured for Ansible
- [ ] Repository secrets configured in Gitea
- [ ] CI/CD pipeline runs successfully end-to-end

### Application
- [ ] Application accessible at https://michaelschiemer.de
- [ ] Health endpoint returns 200 OK
- [ ] Database migrations ran successfully
- [ ] Queue workers processing jobs
- [ ] Logs showing no errors

### Monitoring
- [ ] Portainer shows all containers running
- [ ] Grafana dashboards displaying metrics
- [ ] Prometheus scraping all targets
- [ ] Alerts configured and sending notifications

### Security
- [ ] All secrets encrypted with Ansible Vault
- [ ] SSH keys secured (600 permissions)
- [ ] Registry requires authentication
- [ ] HTTPS enforced on all public endpoints
- [ ] Firewall configured correctly

---

## Troubleshooting

### Gitea Runner Not Registering

**Symptoms**: Runner not appearing in Gitea UI after running `./register.sh`

**Solutions**:
```bash
# Check runner logs
docker compose logs gitea-runner

# Verify registration token is correct
nano .env
# Check GITEA_RUNNER_REGISTRATION_TOKEN

# Unregister and re-register
./unregister.sh
./register.sh
```

### Ansible Connection Failed

**Symptoms**: `Failed to connect to the host via ssh`

**Solutions**:
```bash
# Test SSH manually
ssh -i ~/.ssh/production deploy@94.16.110.151

# Check SSH key permissions
chmod 600 ~/.ssh/production

# Verify SSH key is added to server
ssh-copy-id -i ~/.ssh/production.pub deploy@94.16.110.151
```

### Docker Registry Authentication Failed

**Symptoms**: `unauthorized: authentication required`

**Solutions**:
```bash
# Verify credentials
docker login registry.michaelschiemer.de
# Username: admin
# Password: <your-registry-password>

# Check htpasswd file on server
ssh deploy@94.16.110.151 "cat ~/deployment/stacks/registry/auth/htpasswd"
```

### Deployment Health Check Failed

**Symptoms**: Health check returns 404 or times out

**Solutions**:
```bash
# Check application logs
ssh deploy@94.16.110.151 "docker compose -f ~/application/docker-compose.yml logs app-php"

# Verify application stack is running
ssh deploy@94.16.110.151 "docker ps"

# Check Traefik routing
ssh deploy@94.16.110.151 "docker compose -f ~/deployment/stacks/traefik/docker-compose.yml logs"
```

---

## Next Steps

After successful deployment:

1. **Configure DNS**: Point michaelschiemer.de to 94.16.110.151
2. **SSL Certificates**: Traefik will automatically request Let's Encrypt certificates
3. **Monitoring**: Review Grafana dashboards and setup additional alerts
4. **Backups**: Configure automated database backups
5. **Performance**: Review application performance and optimize
6. **Documentation**: Update team documentation with production procedures

---

## Support Contacts

- **Infrastructure Issues**: Check Portainer logs
- **Deployment Issues**: Review Gitea Actions logs
- **Application Issues**: Check application logs in Portainer
- **Emergency Rollback**: Run `ansible-playbook playbooks/rollback.yml`

---

**Setup Status**: 🚧 In Progress
**Next Action**: Start with Phase 1 - Gitea Runner Setup