michael/michaelschiemer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
6deca7838e59a563db2dd69b3365a28dd64d44bb
michaelschiemer/deployment/SETUP-GUIDE.md

794 lines
19 KiB
Markdown
Raw Blame History

# Production Deployment - Complete Setup Guide
**Status**: 🚧 In Progress
**Last Updated**: 2025-10-30
**Target Server**: 94.16.110.151 (Netcup)
---
## Overview
This guide walks through the complete setup of production deployment from scratch, covering:
1. Gitea Runner (Development Machine)
2. Ansible Vault Secrets
3. Production Server Initial Setup
4. CI/CD Pipeline Testing
5. Monitoring & Health Checks
---
## Prerequisites
**Development Machine:**
- ✅ Docker & Docker Compose installed
- ✅ Ansible installed (`pip install ansible`)
- ✅ SSH key for production server (`~/.ssh/production`)
- ✅ Git SSH key configured (see Phase 0)
- ✅ Access to Gitea admin panel
**Production Server (94.16.110.151):**
- ✅ Docker & Docker Compose installed
- ✅ User `deploy` created with Docker permissions
- ✅ SSH access configured
- ✅ Firewall configured (ports 80, 443, 2222)
---
## Phase 0: Git Repository SSH Access Setup (Development Machine)
### Step 0.1: Generate Git SSH Key
Create a separate SSH key specifically for Git operations (different from the production server SSH key):
```bash
# Generate SSH key for Git
ssh-keygen -t ed25519 -f ~/.ssh/git_michaelschiemer -C "git@michaelschiemer.de" -N ""
# Set correct permissions
chmod 600 ~/.ssh/git_michaelschiemer
chmod 644 ~/.ssh/git_michaelschiemer.pub
```
### Step 0.2: Configure SSH Config
Add Git SSH configuration to `~/.ssh/config`:
```bash
# Edit SSH config
nano ~/.ssh/config
```
Add the following configuration:
```
Host git.michaelschiemer.de
HostName git.michaelschiemer.de
Port 2222
User git
IdentityFile ~/.ssh/git_michaelschiemer
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
```
### Step 0.3: Add Public Key to Gitea
1. Display your public key:
```bash
cat ~/.ssh/git_michaelschiemer.pub
```
2. Copy the output (starts with `ssh-ed25519 ...`)
3. In Gitea:
- Go to **Settings** → **SSH / GPG Keys**
- Click **Add Key**
- Paste the public key
- Click **Add Key**
4. Verify the connection:
```bash
ssh -T git@git.michaelschiemer.de
```
Expected output: `Hi there! You've successfully authenticated...`
### Step 0.4: Update Git Remote (if needed)
If your `origin` remote uses HTTPS, switch it to SSH:
```bash
# Check current remote URL
git remote -v
# Update to SSH
git remote set-url origin git@git.michaelschiemer.de:michael/michaelschiemer.git
# Test push (should work without password prompt)
git push origin main
```
**Note**: This SSH key is separate from the production server SSH key (`~/.ssh/production`). The production key is used for Ansible/server access, while the Git key is only for repository operations.
---
## Phase 1: Gitea Runner Setup (Development Machine)
### Step 1.1: Get Gitea Registration Token
1. Navigate to Gitea admin panel:
```
https://git.michaelschiemer.de/admin/actions/runners
```
2. Click **"Create New Runner"**
3. Copy the registration token (format: `<long-random-string>`)
### Step 1.2: Configure Runner Environment
```bash
cd deployment/gitea-runner
# Copy environment template
cp .env.example .env
# Edit configuration
nano .env
```
**Required Configuration in `.env`:**
```bash
# Gitea Instance URL
GITEA_INSTANCE_URL=https://git.michaelschiemer.de
# Registration Token (from Step 1.1)
GITEA_RUNNER_REGISTRATION_TOKEN=<your-token-from-gitea>
# Runner Name (appears in Gitea UI)
GITEA_RUNNER_NAME=dev-runner-01
# Runner Labels (environments this runner supports)
GITEA_RUNNER_LABELS=ubuntu-latest:docker://node:20-bullseye,ubuntu-22.04:docker://catthehacker/ubuntu:act-22.04
# Runner Capacity (concurrent jobs)
GITEA_RUNNER_CAPACITY=1
# Docker-in-Docker settings
DOCKER_HOST=tcp://docker-dind:2376
DOCKER_TLS_VERIFY=1
```
### Step 1.3: Register and Start Runner
```bash
# Register runner with Gitea
./register.sh
# Expected output:
# ✅ Starting Gitea Runner services...
# ✅ Runner registered successfully
# ✅ Runner is now active
# Verify runner is running
docker compose ps
# Check logs
docker compose logs -f gitea-runner
```
### Step 1.4: Verify Runner in Gitea
1. Go to: https://git.michaelschiemer.de/admin/actions/runners
2. You should see `dev-runner-01` listed as **"Idle"** or **"Active"**
3. Status should be green/online
**✅ Checkpoint**: Runner visible in Gitea UI and showing as "Idle"
---
## Phase 2: Ansible Vault Secrets Setup
### Step 2.1: Create Vault Password
```bash
cd deployment/ansible/secrets
# Create vault password file (gitignored)
echo "your-secure-vault-password-here" > .vault_pass
# Secure the file
chmod 600 .vault_pass
```
**⚠️ IMPORTANT**: Store this vault password in your password manager! You'll need it for all Ansible operations.
### Step 2.2: Create Production Secrets File
```bash
# Copy example template
cp production.vault.yml.example production.vault.yml
# Edit with your actual secrets
nano production.vault.yml
```
**Required Secrets in `production.vault.yml`:**
```yaml
---
# Docker Registry Credentials
docker_registry_user: "admin"
docker_registry_password: "your-registry-password"
# Application Environment Variables
app_key: "base64:generated-32-character-key"
app_env: "production"
app_debug: "false"
# Database Credentials
db_host: "postgres"
db_port: "5432"
db_name: "framework_production"
db_user: "framework_user"
db_password: "your-secure-db-password"
# Redis Configuration
redis_host: "redis"
redis_port: "6379"
redis_password: "your-secure-redis-password"
# Cache Configuration
cache_driver: "redis"
cache_prefix: "framework"
# Queue Configuration
queue_connection: "redis"
queue_name: "default"
# Session Configuration
session_driver: "redis"
session_lifetime: "120"
# Encryption Keys
encryption_key: "base64:your-32-byte-encryption-key"
state_encryption_key: "base64:your-32-byte-state-encryption-key"
# SMTP Configuration (Optional)
mail_mailer: "smtp"
mail_host: "smtp.example.com"
mail_port: "587"
mail_username: "noreply@michaelschiemer.de"
mail_password: "your-smtp-password"
mail_encryption: "tls"
mail_from_address: "noreply@michaelschiemer.de"
mail_from_name: "Framework"
# Admin IPs (comma-separated)
admin_allowed_ips: "127.0.0.1,::1"
# Rate Limiting
rate_limit_enabled: "true"
rate_limit_default: "60"
rate_limit_window: "60"
```
### Step 2.3: Generate Encryption Keys
```bash
# Generate app_key (32 bytes base64)
php -r "echo 'base64:' . base64_encode(random_bytes(32)) . PHP_EOL;"
# Generate encryption_key (32 bytes base64)
php -r "echo 'base64:' . base64_encode(random_bytes(32)) . PHP_EOL;"
# Generate state_encryption_key (32 bytes base64)
php -r "echo 'base64:' . base64_encode(random_bytes(32)) . PHP_EOL;"
# Copy these values into production.vault.yml
```
### Step 2.4: Encrypt Secrets File
```bash
# Encrypt the secrets file
ansible-vault encrypt production.vault.yml \
--vault-password-file .vault_pass
# Verify encryption worked
file production.vault.yml
# Should output: production.vault.yml: ASCII text
# View encrypted content (should show encrypted data)
cat production.vault.yml
# Test decryption (view content)
ansible-vault view production.vault.yml \
--vault-password-file .vault_pass
```
**✅ Checkpoint**: `production.vault.yml` is encrypted and can be decrypted with vault password
---
## Phase 3: Production Server Initial Setup
### Prerequisites
Before running Phase 3, ensure:
- ✅ SSH access to production server configured (`~/.ssh/production`)
- ✅ Repository cloned on production server at `~/deployment/stacks` (or adjust `stacks_base_path` in playbook)
- ✅ Ansible installed on your development machine: `pip install ansible`
- ✅ Ansible collections installed: `ansible-galaxy collection install community.docker`
### Step 3.1: Clone Repository on Production Server (if not already done)
**On Production Server:**
```bash
# SSH to production server
ssh deploy@94.16.110.151
# Clone repository (if not already present)
mkdir -p ~/deployment
cd ~/deployment
git clone git@git.michaelschiemer.de:michael/michaelschiemer.git . || git clone https://git.michaelschiemer.de/michael/michaelschiemer.git .
```
### Step 3.2: Deploy Infrastructure Stacks with Ansible
**On Development Machine:**
```bash
# Navigate to Ansible directory
cd deployment/ansible
# Run infrastructure deployment playbook
ansible-playbook playbooks/setup-infrastructure.yml \
-i inventory/production.yml
# The playbook will:
# 1. Create required Docker networks (traefik-public, app-internal)
# 2. Deploy Traefik (Reverse Proxy & SSL)
# 3. Deploy PostgreSQL (Database)
# 4. Deploy Docker Registry (Private Registry)
# 5. Deploy Gitea (Git Server + PostgreSQL)
# 6. Deploy Monitoring (Portainer + Grafana + Prometheus)
# 7. Wait for all services to be healthy
# 8. Verify accessibility
```
**Expected output:**
- ✅ All stacks deployed successfully
- ✅ All services healthy
- ✅ Gitea accessible at https://git.michaelschiemer.de
**Note:** If monitoring passwords need to be stored in Vault (recommended for production), add them to `secrets/production.vault.yml`:
- `vault_grafana_admin_password`
- `vault_prometheus_password`
Then run the playbook with vault:
```bash
ansible-playbook playbooks/setup-infrastructure.yml \
-i inventory/production.yml \
--vault-password-file secrets/.vault_pass
```
### Step 3.3: Configure Gitea (Manual Step)
1. Access Gitea: https://git.michaelschiemer.de
2. Complete initial setup wizard (first-time only):
- **Database Type**: PostgreSQL
- **Database Host**: `postgres:5432`
- **Database User**: `gitea`
- **Database Password**: `gitea_password` (or check `deployment/stacks/gitea/docker-compose.yml`)
- **Database Name**: `gitea`
- **Admin Account**: Create your admin user
- **Repository Root**: `/data/git/repositories` (default)
3. **Enable Actions** (required for Phase 1):
- Go to **Site Administration** → **Actions**
- Enable **Enable Actions** checkbox
- Save settings
### Step 3.4: Verify Docker Registry
The Ansible playbook automatically creates registry authentication. To retrieve credentials:
```bash
# SSH to production server
ssh deploy@94.16.110.151
# View registry htpasswd (contains username:password hash)
cat ~/deployment/stacks/registry/auth/htpasswd
# The default username is 'admin'
# Password hash can be used to login, or create new user:
cd ~/deployment/stacks/registry
docker compose exec registry htpasswd -Bbn <username> <password> >> auth/htpasswd
docker compose restart registry
# Test login
docker login registry.michaelschiemer.de
# Or if using port:
docker login git.michaelschiemer.de:5000
```
**✅ Checkpoint**: All infrastructure stacks running, Gitea accessible, Actions enabled
---
## Phase 4: Deploy Application Secrets
### Step 4.1: Deploy Secrets to Production
**On Development Machine:**
```bash
cd deployment/ansible
# Test Ansible connectivity
ansible production -m ping
# Deploy secrets to production server
ansible-playbook playbooks/setup-production-secrets.yml \
--vault-password-file secrets/.vault_pass
# Expected output:
# PLAY [Deploy Production Secrets] ***
# TASK [Ensure secrets directory exists] *** ok
# TASK [Deploy environment file] *** changed
# PLAY RECAP *** production: ok=2 changed=1
```
### Step 4.2: Verify Secrets Deployed
```bash
# SSH to production server
ssh deploy@94.16.110.151
# Check secrets directory
ls -la ~/secrets/
# Verify .env.production exists (do NOT cat - contains secrets!)
file ~/secrets/.env.production
# Should output: .env.production: ASCII text
# Check file permissions
stat ~/secrets/.env.production
# Should be 600 (readable only by deploy user)
```
**✅ Checkpoint**: Secrets deployed to production server in ~/secrets/.env.production
---
## Phase 5: Setup Gitea Secrets for CI/CD
### Step 5.1: Configure Repository Secrets
1. Go to repository settings in Gitea:
```
https://git.michaelschiemer.de/<username>/michaelschiemer/settings/secrets
```
2. Add the following secrets:
**REGISTRY_USER**
```
admin
```
**REGISTRY_PASSWORD**
```
<your-registry-password>
```
**SSH_PRIVATE_KEY**
```
<content-of-~/.ssh/production>
```
**ANSIBLE_VAULT_PASSWORD**
```
<your-vault-password-from-step-2.1>
```
### Step 5.2: Verify Secrets in Gitea
1. Check secrets are visible in repository settings
2. Each secret should show "Hidden" value with green checkmark
**✅ Checkpoint**: All required secrets configured in Gitea repository
---
## Phase 6: First Deployment Test
### Step 6.1: Manual Deployment Dry-Run
**On Development Machine:**
```bash
cd deployment/ansible
# Test deployment (check mode - no changes)
ansible-playbook -i inventory/production.yml \
playbooks/deploy-update.yml \
-e "image_tag=test-$(date +%s)" \
-e "git_commit_sha=test123" \
-e "deployment_timestamp=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
-e "docker_registry_username=admin" \
-e "docker_registry_password=your-registry-password" \
--check
# Expected: Should show what would be changed
```
### Step 6.2: Trigger CI/CD Pipeline
**Option A: Push to main branch**
```bash
# Make a small change (add comment to file)
echo "# Deployment test $(date)" >> deployment/DEPLOYMENT_TEST.txt
# Commit and push to main
git add deployment/DEPLOYMENT_TEST.txt
git commit -m "test(deployment): trigger CI/CD pipeline"
git push origin main
```
**Option B: Manual trigger**
1. Go to Gitea repository: Actions tab
2. Select workflow: "Production Deployment Pipeline"
3. Click "Run workflow"
4. Select branch: main
5. Click "Run"
### Step 6.3: Monitor Pipeline Execution
1. Go to: https://git.michaelschiemer.de/<username>/michaelschiemer/actions
2. Find the running workflow
3. Click to view details
4. Monitor each job:
- ✅ Test: Tests & quality checks pass
- ✅ Build: Docker image built and pushed
- ✅ Deploy: Application deployed to production
### Step 6.4: Verify Deployment
```bash
# Test health endpoint
curl -k https://michaelschiemer.de/health
# Expected response:
# {"status":"healthy","timestamp":"2025-10-30T14:30:00Z"}
# Check application logs
ssh deploy@94.16.110.151 "docker compose -f ~/application/docker-compose.yml logs -f app-php"
```
**✅ Checkpoint**: CI/CD pipeline executed successfully, application running on production
---
## Phase 7: Monitoring & Health Checks
### Step 7.1: Access Monitoring Tools
**Portainer**
```
https://portainer.michaelschiemer.de
```
- View all running containers
- Monitor resource usage
- Check logs
**Grafana**
```
https://grafana.michaelschiemer.de
```
- Username: admin
- Password: (set during setup)
- View application metrics
- Setup alerts
**Prometheus**
```
https://prometheus.michaelschiemer.de
```
- Query metrics
- Check targets
- Verify scraping
### Step 7.2: Configure Alerting
**In Grafana:**
1. Go to Alerting > Contact points
2. Add email notification channel
3. Create alert rules:
- High CPU usage (>80% for 5 minutes)
- High memory usage (>80%)
- Application down (health check fails)
- Database connection failures
### Step 7.3: Setup Health Check Monitoring
```bash
# Create cron job on production server
ssh deploy@94.16.110.151
# Add health check script
crontab -e
# Add line:
*/5 * * * * curl -f https://michaelschiemer.de/health || echo "Health check failed" | mail -s "Production Health Check Failed" admin@michaelschiemer.de
```
**✅ Checkpoint**: Monitoring tools accessible, alerts configured
---
## Phase 8: Backup & Rollback Testing
### Step 8.1: Verify Backups
```bash
# SSH to production server
ssh deploy@94.16.110.151
# Check backup directory
ls -lh ~/backups/
# Should see backup folders with timestamps
# Example: 2025-10-30T14-30-00/
```
### Step 8.2: Test Rollback
```bash
# On development machine
cd deployment/ansible
# Rollback to previous version
ansible-playbook -i inventory/production.yml \
playbooks/rollback.yml
# Verify rollback worked
curl -k https://michaelschiemer.de/health
```
**✅ Checkpoint**: Backups created, rollback mechanism tested
---
## Verification Checklist
### Infrastructure
- [ ] Traefik running and routing HTTPS
- [ ] PostgreSQL accessible and accepting connections
- [ ] Docker Registry accessible at git.michaelschiemer.de:5000
- [ ] Gitea accessible at git.michaelschiemer.de
- [ ] Monitoring stack (Portainer, Grafana, Prometheus) running
### Deployment
- [ ] Gitea Runner registered and showing "Idle" in UI
- [ ] Ansible Vault secrets encrypted and deployable
- [ ] SSH access configured for Ansible
- [ ] Repository secrets configured in Gitea
- [ ] CI/CD pipeline runs successfully end-to-end
### Application
- [ ] Application accessible at https://michaelschiemer.de
- [ ] Health endpoint returns 200 OK
- [ ] Database migrations ran successfully
- [ ] Queue workers processing jobs
- [ ] Logs showing no errors
### Monitoring
- [ ] Portainer shows all containers running
- [ ] Grafana dashboards displaying metrics
- [ ] Prometheus scraping all targets
- [ ] Alerts configured and sending notifications
### Security
- [ ] All secrets encrypted with Ansible Vault
- [ ] SSH keys secured (600 permissions)
- [ ] Registry requires authentication
- [ ] HTTPS enforced on all public endpoints
- [ ] Firewall configured correctly
---
## Troubleshooting
### Gitea Runner Not Registering
**Symptoms**: Runner not appearing in Gitea UI after running `./register.sh`
**Solutions**:
```bash
# Check runner logs
docker compose logs gitea-runner
# Verify registration token is correct
nano .env
# Check GITEA_RUNNER_REGISTRATION_TOKEN
# Unregister and re-register
./unregister.sh
./register.sh
```
### Ansible Connection Failed
**Symptoms**: `Failed to connect to the host via ssh`
**Solutions**:
```bash
# Test SSH manually
ssh -i ~/.ssh/production deploy@94.16.110.151
# Check SSH key permissions
chmod 600 ~/.ssh/production
# Verify SSH key is added to server
ssh-copy-id -i ~/.ssh/production.pub deploy@94.16.110.151
```
### Docker Registry Authentication Failed
**Symptoms**: `unauthorized: authentication required`
**Solutions**:
```bash
# Verify credentials
docker login git.michaelschiemer.de:5000
# Username: admin
# Password: <your-registry-password>
# Check htpasswd file on server
ssh deploy@94.16.110.151 "cat ~/deployment/stacks/registry/auth/htpasswd"
```
### Deployment Health Check Failed
**Symptoms**: Health check returns 404 or times out
**Solutions**:
```bash
# Check application logs
ssh deploy@94.16.110.151 "docker compose -f ~/application/docker-compose.yml logs app-php"
# Verify application stack is running
ssh deploy@94.16.110.151 "docker ps"
# Check Traefik routing
ssh deploy@94.16.110.151 "docker compose -f ~/deployment/stacks/traefik/docker-compose.yml logs"
```
---
## Next Steps
After successful deployment:
1. **Configure DNS**: Point michaelschiemer.de to 94.16.110.151
2. **SSL Certificates**: Traefik will automatically request Let's Encrypt certificates
3. **Monitoring**: Review Grafana dashboards and setup additional alerts
4. **Backups**: Configure automated database backups
5. **Performance**: Review application performance and optimize
6. **Documentation**: Update team documentation with production procedures
---
## Support Contacts
- **Infrastructure Issues**: Check Portainer logs
- **Deployment Issues**: Review Gitea Actions logs
- **Application Issues**: Check application logs in Portainer
- **Emergency Rollback**: Run `ansible-playbook playbooks/rollback.yml`
---
**Setup Status**: 🚧 In Progress
**Next Action**: Start with Phase 1 - Gitea Runner Setup
Reference in New Issue View Git Blame Copy Permalink