feat: CI/CD pipeline setup complete - Ansible playbooks updated, secrets configured, workflow ready

2025-10-31 01:39:24 +01:00
parent 55c04e4fd0
commit e26eb2aa12
601 changed files with 44184 additions and 32477 deletions
--- a/deployment/stacks/postgresql/README.md
+++ b/deployment/stacks/postgresql/README.md
@@ -0,0 +1,681 @@
+# PostgreSQL Stack - Production Database with Automated Backups
+
+## Overview
+
+Production-ready PostgreSQL 16 database with automated backup system and performance optimization.
+
+**Features**:
+- PostgreSQL 16 Alpine (lightweight, secure)
+- Automated daily backups with configurable retention
+- Performance-optimized configuration (2GB memory allocation)
+- Health checks and automatic recovery
+- Persistent storage with named volumes
+- Isolated app-internal network
+- Resource limits for stability
+
+## Services
+
+- **postgres** - PostgreSQL 16 database server
+- **postgres-backup** - Automated backup service with cron scheduling
+
+## Prerequisites
+
+1. **Traefik Stack Running**
+   ```bash
+   cd ../traefik
+   docker compose up -d
+   ```
+
+2. **App-Internal Network Created**
+   ```bash
+   docker network create app-internal
+   ```
+   (Created automatically by Stack 4 - Application)
+
+## Configuration
+
+### 1. Create Environment File
+
+```bash
+cp .env.example .env
+```
+
+### 2. Generate Secure Password
+
+```bash
+openssl rand -base64 32
+```
+
+Update `.env`:
+```env
+POSTGRES_PASSWORD=<generated-password>
+```
+
+### 3. Review Configuration
+
+**Database Settings** (`.env`):
+- `POSTGRES_DB` - Database name (default: michaelschiemer)
+- `POSTGRES_USER` - Database user (default: postgres)
+- `POSTGRES_PASSWORD` - Database password (REQUIRED)
+
+**Backup Settings** (`.env`):
+- `BACKUP_RETENTION_DAYS` - Keep backups for N days (default: 7)
+- `BACKUP_SCHEDULE` - Cron expression (default: `0 2 * * *` = 2 AM daily)
+
+**Performance Tuning** (`conf.d/postgresql.conf`):
+- Optimized for 2GB memory allocation
+- Connection pooling (max 100 connections)
+- Write-ahead logging for reliability
+- Query logging for slow queries (>1s)
+- Parallel query execution enabled
+
+## Deployment
+
+### Initial Setup
+
+```bash
+# Create environment file
+cp .env.example .env
+
+# Generate and set password
+openssl rand -base64 32
+# Update POSTGRES_PASSWORD in .env
+
+# Ensure app-internal network exists
+docker network inspect app-internal || docker network create app-internal
+
+# Start services
+docker compose up -d
+
+# Check logs
+docker compose logs -f
+
+# Verify health
+docker compose ps
+```
+
+### Verify Deployment
+
+```bash
+# Check PostgreSQL is running
+docker exec postgres pg_isready -U postgres -d michaelschiemer
+
+# Expected: postgres:5432 - accepting connections
+
+# Check backup service
+docker compose logs postgres-backup
+
+# Expected: Initial backup completed successfully
+```
+
+## Usage
+
+### Database Access
+
+#### From Host Machine
+
+```bash
+# Connect to database
+docker exec -it postgres psql -U postgres -d michaelschiemer
+
+# Run SQL query
+docker exec postgres psql -U postgres -d michaelschiemer -c "SELECT version();"
+```
+
+#### From Application Container
+
+```bash
+# Connection string format
+postgresql://postgres:password@postgres:5432/michaelschiemer
+
+# Example with environment variables (Stack 4 - Application)
+DB_HOST=postgres
+DB_PORT=5432
+DB_NAME=michaelschiemer
+DB_USER=postgres
+DB_PASS=<same-as-POSTGRES_PASSWORD>
+```
+
+### Backup Management
+
+#### Manual Backup
+
+```bash
+# Trigger manual backup
+docker exec postgres-backup /scripts/backup.sh
+
+# List backups
+ls -lh backups/
+
+# Example output:
+# postgres_michaelschiemer_20250130_020000.sql.gz
+# postgres_michaelschiemer_20250131_020000.sql.gz
+```
+
+#### Restore from Backup
+
+```bash
+# List available backups
+docker exec postgres-backup ls -lh /backups
+
+# Restore specific backup
+docker exec -it postgres-backup /scripts/restore.sh /backups/postgres_michaelschiemer_20250130_020000.sql.gz
+
+# ⚠️ WARNING: This will DROP and RECREATE the database!
+# Confirm after 10 second countdown
+```
+
+#### Download Backup
+
+```bash
+# Copy backup to host
+docker cp postgres-backup:/backups/postgres_michaelschiemer_20250130_020000.sql.gz ./local-backup.sql.gz
+
+# Extract and inspect
+gunzip -c local-backup.sql.gz | less
+```
+
+### Database Maintenance
+
+#### Vacuum and Analyze
+
+```bash
+# Full vacuum (recommended weekly)
+docker exec postgres psql -U postgres -d michaelschiemer -c "VACUUM FULL ANALYZE;"
+
+# Quick vacuum (automatic, but can run manually)
+docker exec postgres psql -U postgres -d michaelschiemer -c "VACUUM ANALYZE;"
+```
+
+#### Check Database Size
+
+```bash
+docker exec postgres psql -U postgres -d michaelschiemer -c "
+SELECT
+  pg_size_pretty(pg_database_size('michaelschiemer')) as db_size,
+  pg_size_pretty(pg_total_relation_size('users')) as users_table_size;
+"
+```
+
+#### Connection Statistics
+
+```bash
+docker exec postgres psql -U postgres -d michaelschiemer -c "
+SELECT
+  datname,
+  numbackends as connections,
+  xact_commit as commits,
+  xact_rollback as rollbacks
+FROM pg_stat_database
+WHERE datname = 'michaelschiemer';
+"
+```
+
+### Performance Monitoring
+
+#### Active Queries
+
+```bash
+docker exec postgres psql -U postgres -d michaelschiemer -c "
+SELECT
+  pid,
+  usename,
+  application_name,
+  state,
+  query_start,
+  query
+FROM pg_stat_activity
+WHERE state != 'idle'
+ORDER BY query_start;
+"
+```
+
+#### Slow Queries
+
+```bash
+# Check PostgreSQL logs for slow queries (>1s)
+docker exec postgres tail -f /var/lib/postgresql/data/pgdata/log/postgresql-*.log
+```
+
+#### Index Usage
+
+```bash
+docker exec postgres psql -U postgres -d michaelschiemer -c "
+SELECT
+  schemaname,
+  tablename,
+  indexname,
+  idx_scan as index_scans,
+  idx_tup_read as tuples_read
+FROM pg_stat_user_indexes
+ORDER BY idx_scan DESC;
+"
+```
+
+## Integration with Other Stacks
+
+### Stack 4: Application
+
+Update `deployment/stacks/application/.env`:
+
+```env
+# Database Configuration
+DB_HOST=postgres
+DB_PORT=5432
+DB_NAME=michaelschiemer
+DB_USER=postgres
+DB_PASS=<same-as-postgres-stack-password>
+```
+
+**Connection Test** from Application:
+```bash
+# From app container
+docker exec app php -r "
+\$dsn = 'pgsql:host=postgres;port=5432;dbname=michaelschiemer';
+\$pdo = new PDO(\$dsn, 'postgres', getenv('DB_PASS'));
+echo 'Connection successful: ' . \$pdo->query('SELECT version()')->fetchColumn();
+"
+```
+
+### Stack 2: Gitea (Optional PostgreSQL Backend)
+
+If migrating Gitea from MySQL to PostgreSQL:
+
+```env
+# In deployment/stacks/gitea/.env
+DB_TYPE=postgres
+DB_HOST=postgres
+DB_NAME=gitea
+DB_USER=postgres
+DB_PASS=<same-password>
+```
+
+**Note**: Requires creating separate `gitea` database:
+```bash
+docker exec postgres psql -U postgres -c "CREATE DATABASE gitea;"
+```
+
+## Backup & Recovery
+
+### Automated Backup Strategy
+
+**Schedule**: Daily at 2:00 AM (configurable via `BACKUP_SCHEDULE`)
+
+**Retention**: 7 days (configurable via `BACKUP_RETENTION_DAYS`)
+
+**Location**: `./backups/` directory on host
+
+**Format**: Compressed SQL dumps (`postgres_<dbname>_<timestamp>.sql.gz`)
+
+### Manual Backup Workflow
+
+```bash
+# 1. Create manual backup
+docker exec postgres-backup /scripts/backup.sh
+
+# 2. Verify backup
+ls -lh backups/
+
+# 3. Test backup integrity (optional)
+gunzip -t backups/postgres_michaelschiemer_20250130_020000.sql.gz
+```
+
+### Disaster Recovery
+
+#### Scenario: Complete Database Loss
+
+```bash
+# 1. Stop application to prevent writes
+cd ../application
+docker compose stop
+
+# 2. Remove corrupted database
+cd ../postgresql
+docker compose down
+docker volume rm postgres-data
+
+# 3. Recreate database
+docker compose up -d
+
+# 4. Wait for PostgreSQL to initialize
+docker compose logs -f postgres
+
+# 5. Restore from latest backup
+docker exec -it postgres-backup /scripts/restore.sh /backups/postgres_michaelschiemer_<latest>.sql.gz
+
+# 6. Verify restoration
+docker exec postgres psql -U postgres -d michaelschiemer -c "\dt"
+
+# 7. Restart application
+cd ../application
+docker compose start
+```
+
+#### Scenario: Point-in-Time Recovery
+
+```bash
+# 1. List available backups
+docker exec postgres-backup ls -lh /backups
+
+# 2. Choose backup timestamp
+# postgres_michaelschiemer_20250130_143000.sql.gz
+
+# 3. Restore to that point
+docker exec -it postgres-backup /scripts/restore.sh /backups/postgres_michaelschiemer_20250130_143000.sql.gz
+```
+
+### Off-site Backup
+
+**Recommended**: Copy backups to external storage
+
+```bash
+#!/bin/bash
+# backup-offsite.sh - Run daily after local backup
+
+BACKUP_DIR="./backups"
+REMOTE_HOST="backup-server.example.com"
+REMOTE_PATH="/backups/michaelschiemer/postgresql"
+
+# Sync backups to remote server
+rsync -avz --delete \
+  "${BACKUP_DIR}/" \
+  "${REMOTE_HOST}:${REMOTE_PATH}/"
+
+echo "✅ Off-site backup completed"
+```
+
+**Alternative: S3 Upload**
+```bash
+# Using AWS CLI
+aws s3 sync ./backups/ s3://my-backup-bucket/postgresql/ --delete
+```
+
+## Security
+
+### Connection Security
+
+**Network Isolation**:
+- PostgreSQL only accessible via `app-internal` network
+- No external ports exposed
+- Service-to-service communication only
+
+**Authentication**:
+- Strong password required (generated with `openssl rand -base64 32`)
+- No default passwords
+- Password stored in environment variables only
+
+### Backup Security
+
+**Encryption** (recommended for production):
+```bash
+# Encrypt backup before off-site storage
+gpg --symmetric --cipher-algo AES256 backups/postgres_michaelschiemer_*.sql.gz
+
+# Decrypt when needed
+gpg --decrypt backups/postgres_michaelschiemer_*.sql.gz.gpg | gunzip | psql
+```
+
+**Access Control**:
+- Backup directory mounted as read-only in other containers
+- Backup service has write access only
+- Host filesystem permissions: `chmod 700 backups/`
+
+### Update Security
+
+```bash
+# Update PostgreSQL image
+docker compose pull
+
+# Recreate containers with new image
+docker compose up -d
+
+# Verify version
+docker exec postgres psql -U postgres -c "SELECT version();"
+```
+
+## Monitoring
+
+### Health Checks
+
+```bash
+# Check service health
+docker compose ps
+
+# Expected: Both services "healthy"
+
+# Manual health check
+docker exec postgres pg_isready -U postgres -d michaelschiemer
+
+# Check backup service
+docker compose logs postgres-backup | grep "✅ Backup completed"
+```
+
+### Resource Usage
+
+```bash
+# Database container stats
+docker stats postgres --no-stream
+
+# Expected:
+# - Memory: ~200-800MB (under 2GB limit)
+# - CPU: <50% sustained
+
+# Disk usage
+docker exec postgres du -sh /var/lib/postgresql/data
+```
+
+### Logs
+
+```bash
+# PostgreSQL logs
+docker compose logs postgres
+
+# Backup logs
+docker compose logs postgres-backup
+
+# Real-time monitoring
+docker compose logs -f
+
+# PostgreSQL server logs (inside container)
+docker exec postgres tail -f /var/lib/postgresql/data/pgdata/log/postgresql-*.log
+```
+
+### Alerts
+
+**Recommended Monitoring**:
+- Backup success/failure notifications
+- Disk space warnings (>80% full)
+- Connection count monitoring
+- Slow query alerts
+- Replication lag (if using replication)
+
+## Troubleshooting
+
+### Database Won't Start
+
+```bash
+# Check logs
+docker compose logs postgres
+
+# Common issues:
+# 1. Invalid configuration
+docker exec postgres postgres --check
+
+# 2. Corrupted data directory
+docker compose down
+docker volume rm postgres-data
+docker compose up -d
+
+# 3. Permission issues
+docker exec postgres ls -la /var/lib/postgresql/data
+```
+
+### Backup Failures
+
+```bash
+# Check backup service logs
+docker compose logs postgres-backup
+
+# Common issues:
+# 1. Disk full
+df -h
+
+# 2. Connection to PostgreSQL failed
+docker exec postgres-backup pg_isready -h postgres -U postgres
+
+# 3. Manual backup test
+docker exec postgres-backup /scripts/backup.sh
+```
+
+### Connection Refused from Application
+
+```bash
+# 1. Check PostgreSQL is running
+docker compose ps postgres
+
+# 2. Verify network
+docker network inspect app-internal | grep postgres
+
+# 3. Test connection
+docker exec app nc -zv postgres 5432
+
+# 4. Check credentials
+docker exec app printenv | grep DB_
+```
+
+### Slow Queries
+
+```bash
+# Enable extended logging
+docker exec postgres psql -U postgres -c "ALTER SYSTEM SET log_min_duration_statement = 500;"
+docker compose restart postgres
+
+# Check for missing indexes
+docker exec postgres psql -U postgres -d michaelschiemer -c "
+SELECT
+  schemaname,
+  tablename,
+  attname,
+  n_distinct,
+  correlation
+FROM pg_stats
+WHERE schemaname = 'public'
+ORDER BY correlation;
+"
+```
+
+### Out of Disk Space
+
+```bash
+# Check disk usage
+df -h
+
+# Check database size
+docker exec postgres psql -U postgres -d michaelschiemer -c "
+SELECT pg_size_pretty(pg_database_size('michaelschiemer'));
+"
+
+# Vacuum to reclaim space
+docker exec postgres psql -U postgres -d michaelschiemer -c "VACUUM FULL;"
+
+# Clean old backups manually
+find ./backups -name "*.sql.gz" -mtime +30 -delete
+```
+
+## Performance Tuning
+
+### Current Configuration (2GB Memory)
+
+**`conf.d/postgresql.conf`** optimized for:
+- **Memory**: 2GB allocated (512MB shared buffers, 1.5GB effective cache)
+- **Connections**: 100 max connections
+- **Workers**: 4 parallel workers
+- **Checkpoint**: 2GB max WAL size
+
+### Scaling Up (4GB+ Memory)
+
+```ini
+# conf.d/postgresql.conf
+shared_buffers = 1GB           # 25% of RAM
+effective_cache_size = 3GB     # 75% of RAM
+maintenance_work_mem = 256MB
+work_mem = 10MB
+max_connections = 200
+max_parallel_workers = 8
+```
+
+### Query Optimization
+
+```bash
+# Analyze query performance
+docker exec postgres psql -U postgres -d michaelschiemer -c "
+EXPLAIN ANALYZE
+SELECT * FROM users WHERE email = 'test@example.com';
+"
+
+# Create index for frequently queried columns
+docker exec postgres psql -U postgres -d michaelschiemer -c "
+CREATE INDEX idx_users_email ON users(email);
+"
+```
+
+### Connection Pooling
+
+**Recommended**: Use PgBouncer for connection pooling in high-traffic scenarios
+
+```yaml
+# Add to docker-compose.yml
+pgbouncer:
+  image: pgbouncer/pgbouncer:latest
+  environment:
+    - DATABASES_HOST=postgres
+    - DATABASES_PORT=5432
+    - DATABASES_DBNAME=michaelschiemer
+    - PGBOUNCER_POOL_MODE=transaction
+    - PGBOUNCER_MAX_CLIENT_CONN=1000
+    - PGBOUNCER_DEFAULT_POOL_SIZE=25
+```
+
+## Upgrading PostgreSQL
+
+### Minor Version Upgrade (e.g., 16.0 → 16.1)
+
+```bash
+# Pull latest 16.x image
+docker compose pull
+
+# Recreate container
+docker compose up -d
+
+# Verify version
+docker exec postgres psql -U postgres -c "SELECT version();"
+```
+
+### Major Version Upgrade (e.g., 16 → 17)
+
+```bash
+# 1. Create full backup
+docker exec postgres-backup /scripts/backup.sh
+
+# 2. Stop services
+docker compose down
+
+# 3. Update docker-compose.yml
+# Change: postgres:16-alpine → postgres:17-alpine
+
+# 4. Remove old data volume
+docker volume rm postgres-data
+
+# 5. Start new version
+docker compose up -d
+
+# 6. Restore data
+docker exec -it postgres-backup /scripts/restore.sh /backups/postgres_michaelschiemer_<latest>.sql.gz
+```
+
+## Additional Resources
+
+- **PostgreSQL Documentation**: https://www.postgresql.org/docs/16/
+- **Performance Tuning**: https://wiki.postgresql.org/wiki/Performance_Optimization
+- **Backup Best Practices**: https://www.postgresql.org/docs/16/backup.html
+- **Security Hardening**: https://www.postgresql.org/docs/16/security.html