Files

Michael Schiemer e26eb2aa12 feat: CI/CD pipeline setup complete - Ansible playbooks updated, secrets configured, workflow ready

2025-10-31 01:39:24 +01:00

14 KiB

Raw Blame History

PostgreSQL Stack - Production Database with Automated Backups

Overview

Production-ready PostgreSQL 16 database with automated backup system and performance optimization.

Features:

PostgreSQL 16 Alpine (lightweight, secure)
Automated daily backups with configurable retention
Performance-optimized configuration (2GB memory allocation)
Health checks and automatic recovery
Persistent storage with named volumes
Isolated app-internal network
Resource limits for stability

Services

postgres - PostgreSQL 16 database server
postgres-backup - Automated backup service with cron scheduling

Prerequisites

Traefik Stack Running
```
cd ../traefik
docker compose up -d
```
App-Internal Network Created
```
docker network create app-internal
```
(Created automatically by Stack 4 - Application)

Configuration

1. Create Environment File

cp .env.example .env

2. Generate Secure Password

openssl rand -base64 32

Update .env:

POSTGRES_PASSWORD=<generated-password>

3. Review Configuration

Database Settings (.env):

POSTGRES_DB - Database name (default: michaelschiemer)
POSTGRES_USER - Database user (default: postgres)
POSTGRES_PASSWORD - Database password (REQUIRED)

Backup Settings (.env):

BACKUP_RETENTION_DAYS - Keep backups for N days (default: 7)
BACKUP_SCHEDULE - Cron expression (default: 0 2 * * * = 2 AM daily)

Performance Tuning (conf.d/postgresql.conf):

Optimized for 2GB memory allocation
Connection pooling (max 100 connections)
Write-ahead logging for reliability
Query logging for slow queries (>1s)
Parallel query execution enabled

Deployment

Initial Setup

# Create environment file
cp .env.example .env

# Generate and set password
openssl rand -base64 32
# Update POSTGRES_PASSWORD in .env

# Ensure app-internal network exists
docker network inspect app-internal || docker network create app-internal

# Start services
docker compose up -d

# Check logs
docker compose logs -f

# Verify health
docker compose ps

Verify Deployment

# Check PostgreSQL is running
docker exec postgres pg_isready -U postgres -d michaelschiemer

# Expected: postgres:5432 - accepting connections

# Check backup service
docker compose logs postgres-backup

# Expected: Initial backup completed successfully

Usage

Database Access

From Host Machine

# Connect to database
docker exec -it postgres psql -U postgres -d michaelschiemer

# Run SQL query
docker exec postgres psql -U postgres -d michaelschiemer -c "SELECT version();"

From Application Container

# Connection string format
postgresql://postgres:password@postgres:5432/michaelschiemer

# Example with environment variables (Stack 4 - Application)
DB_HOST=postgres
DB_PORT=5432
DB_NAME=michaelschiemer
DB_USER=postgres
DB_PASS=<same-as-POSTGRES_PASSWORD>

Backup Management

Manual Backup

# Trigger manual backup
docker exec postgres-backup /scripts/backup.sh

# List backups
ls -lh backups/

# Example output:
# postgres_michaelschiemer_20250130_020000.sql.gz
# postgres_michaelschiemer_20250131_020000.sql.gz

Restore from Backup

# List available backups
docker exec postgres-backup ls -lh /backups

# Restore specific backup
docker exec -it postgres-backup /scripts/restore.sh /backups/postgres_michaelschiemer_20250130_020000.sql.gz

# ⚠️ WARNING: This will DROP and RECREATE the database!
# Confirm after 10 second countdown

Download Backup

# Copy backup to host
docker cp postgres-backup:/backups/postgres_michaelschiemer_20250130_020000.sql.gz ./local-backup.sql.gz

# Extract and inspect
gunzip -c local-backup.sql.gz | less

Database Maintenance

Vacuum and Analyze

# Full vacuum (recommended weekly)
docker exec postgres psql -U postgres -d michaelschiemer -c "VACUUM FULL ANALYZE;"

# Quick vacuum (automatic, but can run manually)
docker exec postgres psql -U postgres -d michaelschiemer -c "VACUUM ANALYZE;"

Check Database Size

docker exec postgres psql -U postgres -d michaelschiemer -c "
SELECT
  pg_size_pretty(pg_database_size('michaelschiemer')) as db_size,
  pg_size_pretty(pg_total_relation_size('users')) as users_table_size;
"

Connection Statistics

docker exec postgres psql -U postgres -d michaelschiemer -c "
SELECT
  datname,
  numbackends as connections,
  xact_commit as commits,
  xact_rollback as rollbacks
FROM pg_stat_database
WHERE datname = 'michaelschiemer';
"

Performance Monitoring

Active Queries

docker exec postgres psql -U postgres -d michaelschiemer -c "
SELECT
  pid,
  usename,
  application_name,
  state,
  query_start,
  query
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY query_start;
"

Slow Queries

# Check PostgreSQL logs for slow queries (>1s)
docker exec postgres tail -f /var/lib/postgresql/data/pgdata/log/postgresql-*.log

Index Usage

docker exec postgres psql -U postgres -d michaelschiemer -c "
SELECT
  schemaname,
  tablename,
  indexname,
  idx_scan as index_scans,
  idx_tup_read as tuples_read
FROM pg_stat_user_indexes
ORDER BY idx_scan DESC;
"

Integration with Other Stacks

Stack 4: Application

Update deployment/stacks/application/.env:

# Database Configuration
DB_HOST=postgres
DB_PORT=5432
DB_NAME=michaelschiemer
DB_USER=postgres
DB_PASS=<same-as-postgres-stack-password>

Connection Test from Application:

# From app container
docker exec app php -r "
\$dsn = 'pgsql:host=postgres;port=5432;dbname=michaelschiemer';
\$pdo = new PDO(\$dsn, 'postgres', getenv('DB_PASS'));
echo 'Connection successful: ' . \$pdo->query('SELECT version()')->fetchColumn();
"

Stack 2: Gitea (Optional PostgreSQL Backend)

If migrating Gitea from MySQL to PostgreSQL:

# In deployment/stacks/gitea/.env
DB_TYPE=postgres
DB_HOST=postgres
DB_NAME=gitea
DB_USER=postgres
DB_PASS=<same-password>

Note: Requires creating separate gitea database:

docker exec postgres psql -U postgres -c "CREATE DATABASE gitea;"

Backup & Recovery

Automated Backup Strategy

Schedule: Daily at 2:00 AM (configurable via BACKUP_SCHEDULE)

Retention: 7 days (configurable via BACKUP_RETENTION_DAYS)

Location: ./backups/ directory on host

Format: Compressed SQL dumps (postgres_<dbname>_<timestamp>.sql.gz)

Manual Backup Workflow

# 1. Create manual backup
docker exec postgres-backup /scripts/backup.sh

# 2. Verify backup
ls -lh backups/

# 3. Test backup integrity (optional)
gunzip -t backups/postgres_michaelschiemer_20250130_020000.sql.gz

Disaster Recovery

Scenario: Complete Database Loss

# 1. Stop application to prevent writes
cd ../application
docker compose stop

# 2. Remove corrupted database
cd ../postgresql
docker compose down
docker volume rm postgres-data

# 3. Recreate database
docker compose up -d

# 4. Wait for PostgreSQL to initialize
docker compose logs -f postgres

# 5. Restore from latest backup
docker exec -it postgres-backup /scripts/restore.sh /backups/postgres_michaelschiemer_<latest>.sql.gz

# 6. Verify restoration
docker exec postgres psql -U postgres -d michaelschiemer -c "\dt"

# 7. Restart application
cd ../application
docker compose start

Scenario: Point-in-Time Recovery

# 1. List available backups
docker exec postgres-backup ls -lh /backups

# 2. Choose backup timestamp
# postgres_michaelschiemer_20250130_143000.sql.gz

# 3. Restore to that point
docker exec -it postgres-backup /scripts/restore.sh /backups/postgres_michaelschiemer_20250130_143000.sql.gz

Off-site Backup

Recommended: Copy backups to external storage

#!/bin/bash
# backup-offsite.sh - Run daily after local backup

BACKUP_DIR="./backups"
REMOTE_HOST="backup-server.example.com"
REMOTE_PATH="/backups/michaelschiemer/postgresql"

# Sync backups to remote server
rsync -avz --delete \
  "${BACKUP_DIR}/" \
  "${REMOTE_HOST}:${REMOTE_PATH}/"

echo "✅ Off-site backup completed"

Alternative: S3 Upload

# Using AWS CLI
aws s3 sync ./backups/ s3://my-backup-bucket/postgresql/ --delete

Security

Connection Security

Network Isolation:

PostgreSQL only accessible via app-internal network
No external ports exposed
Service-to-service communication only

Authentication:

Strong password required (generated with openssl rand -base64 32)
No default passwords
Password stored in environment variables only

Backup Security

Encryption (recommended for production):

# Encrypt backup before off-site storage
gpg --symmetric --cipher-algo AES256 backups/postgres_michaelschiemer_*.sql.gz

# Decrypt when needed
gpg --decrypt backups/postgres_michaelschiemer_*.sql.gz.gpg | gunzip | psql

Access Control:

Backup directory mounted as read-only in other containers
Backup service has write access only
Host filesystem permissions: chmod 700 backups/

Update Security

# Update PostgreSQL image
docker compose pull

# Recreate containers with new image
docker compose up -d

# Verify version
docker exec postgres psql -U postgres -c "SELECT version();"

Monitoring

Health Checks

# Check service health
docker compose ps

# Expected: Both services "healthy"

# Manual health check
docker exec postgres pg_isready -U postgres -d michaelschiemer

# Check backup service
docker compose logs postgres-backup | grep "✅ Backup completed"

Resource Usage

# Database container stats
docker stats postgres --no-stream

# Expected:
# - Memory: ~200-800MB (under 2GB limit)
# - CPU: <50% sustained

# Disk usage
docker exec postgres du -sh /var/lib/postgresql/data

Logs

# PostgreSQL logs
docker compose logs postgres

# Backup logs
docker compose logs postgres-backup

# Real-time monitoring
docker compose logs -f

# PostgreSQL server logs (inside container)
docker exec postgres tail -f /var/lib/postgresql/data/pgdata/log/postgresql-*.log

Alerts

Recommended Monitoring:

Backup success/failure notifications
Disk space warnings (>80% full)
Connection count monitoring
Slow query alerts
Replication lag (if using replication)

Troubleshooting

Database Won't Start

# Check logs
docker compose logs postgres

# Common issues:
# 1. Invalid configuration
docker exec postgres postgres --check

# 2. Corrupted data directory
docker compose down
docker volume rm postgres-data
docker compose up -d

# 3. Permission issues
docker exec postgres ls -la /var/lib/postgresql/data

Backup Failures

# Check backup service logs
docker compose logs postgres-backup

# Common issues:
# 1. Disk full
df -h

# 2. Connection to PostgreSQL failed
docker exec postgres-backup pg_isready -h postgres -U postgres

# 3. Manual backup test
docker exec postgres-backup /scripts/backup.sh

Connection Refused from Application

# 1. Check PostgreSQL is running
docker compose ps postgres

# 2. Verify network
docker network inspect app-internal | grep postgres

# 3. Test connection
docker exec app nc -zv postgres 5432

# 4. Check credentials
docker exec app printenv | grep DB_

Slow Queries

# Enable extended logging
docker exec postgres psql -U postgres -c "ALTER SYSTEM SET log_min_duration_statement = 500;"
docker compose restart postgres

# Check for missing indexes
docker exec postgres psql -U postgres -d michaelschiemer -c "
SELECT
  schemaname,
  tablename,
  attname,
  n_distinct,
  correlation
FROM pg_stats
WHERE schemaname = 'public'
ORDER BY correlation;
"

Out of Disk Space

# Check disk usage
df -h

# Check database size
docker exec postgres psql -U postgres -d michaelschiemer -c "
SELECT pg_size_pretty(pg_database_size('michaelschiemer'));
"

# Vacuum to reclaim space
docker exec postgres psql -U postgres -d michaelschiemer -c "VACUUM FULL;"

# Clean old backups manually
find ./backups -name "*.sql.gz" -mtime +30 -delete

Performance Tuning

Current Configuration (2GB Memory)

conf.d/postgresql.conf optimized for:

Memory: 2GB allocated (512MB shared buffers, 1.5GB effective cache)
Connections: 100 max connections
Workers: 4 parallel workers
Checkpoint: 2GB max WAL size

Scaling Up (4GB+ Memory)

# conf.d/postgresql.conf
shared_buffers = 1GB           # 25% of RAM
effective_cache_size = 3GB     # 75% of RAM
maintenance_work_mem = 256MB
work_mem = 10MB
max_connections = 200
max_parallel_workers = 8

Query Optimization

# Analyze query performance
docker exec postgres psql -U postgres -d michaelschiemer -c "
EXPLAIN ANALYZE
SELECT * FROM users WHERE email = 'test@example.com';
"

# Create index for frequently queried columns
docker exec postgres psql -U postgres -d michaelschiemer -c "
CREATE INDEX idx_users_email ON users(email);
"

Connection Pooling

Recommended: Use PgBouncer for connection pooling in high-traffic scenarios

# Add to docker-compose.yml
pgbouncer:
  image: pgbouncer/pgbouncer:latest
  environment:
    - DATABASES_HOST=postgres
    - DATABASES_PORT=5432
    - DATABASES_DBNAME=michaelschiemer
    - PGBOUNCER_POOL_MODE=transaction
    - PGBOUNCER_MAX_CLIENT_CONN=1000
    - PGBOUNCER_DEFAULT_POOL_SIZE=25

Upgrading PostgreSQL

Minor Version Upgrade (e.g., 16.0 → 16.1)

# Pull latest 16.x image
docker compose pull

# Recreate container
docker compose up -d

# Verify version
docker exec postgres psql -U postgres -c "SELECT version();"

Major Version Upgrade (e.g., 16 → 17)

# 1. Create full backup
docker exec postgres-backup /scripts/backup.sh

# 2. Stop services
docker compose down

# 3. Update docker-compose.yml
# Change: postgres:16-alpine → postgres:17-alpine

# 4. Remove old data volume
docker volume rm postgres-data

# 5. Start new version
docker compose up -d

# 6. Restore data
docker exec -it postgres-backup /scripts/restore.sh /backups/postgres_michaelschiemer_<latest>.sql.gz

Additional Resources

PostgreSQL Documentation: https://www.postgresql.org/docs/16/
Performance Tuning: https://wiki.postgresql.org/wiki/Performance_Optimization
Backup Best Practices: https://www.postgresql.org/docs/16/backup.html
Security Hardening: https://www.postgresql.org/docs/16/security.html

14 KiB Raw Blame History

PostgreSQL Stack - Production Database with Automated Backups

Overview

Services

Prerequisites

Configuration

1. Create Environment File

2. Generate Secure Password

3. Review Configuration

Deployment

Initial Setup

Verify Deployment

Usage

Database Access

From Host Machine

From Application Container

Backup Management

Manual Backup

Restore from Backup

Download Backup

Database Maintenance

Vacuum and Analyze

Check Database Size

Connection Statistics

Performance Monitoring

Active Queries

Slow Queries

Index Usage

Integration with Other Stacks

Stack 4: Application

Stack 2: Gitea (Optional PostgreSQL Backend)

Backup & Recovery

Automated Backup Strategy

Manual Backup Workflow

Disaster Recovery

Scenario: Complete Database Loss

Scenario: Point-in-Time Recovery

Off-site Backup

Security

Connection Security

Backup Security

Update Security

Monitoring

Health Checks

Resource Usage

Logs

Alerts

Troubleshooting

Database Won't Start

Backup Failures

Connection Refused from Application

Slow Queries

Out of Disk Space

Performance Tuning

Current Configuration (2GB Memory)

Scaling Up (4GB+ Memory)

Query Optimization

Connection Pooling

Upgrading PostgreSQL

Minor Version Upgrade (e.g., 16.0 → 16.1)

Major Version Upgrade (e.g., 16 → 17)

Additional Resources

14 KiB

Raw Blame History