michaelschiemer/deployment/docs/guides/migrate-to-separate-databases.md

Migration Guide: Separate Database Stacks

Overview

This guide describes how to migrate from a shared PostgreSQL stack to separate database stacks for Production and Staging.

Current State (Before Migration)

• Single PostgreSQL stack: deployment/stacks/postgresql/
• Both Production and Staging connect to the same database instance
• Database name: michaelschiemer (Production) and michaelschiemer_staging (Staging) in same instance

Target State (After Migration)

• Production Database: deployment/stacks/postgresql-production/

  • Container: postgres-production
  • Network: postgres-production-internal
  • Database: michaelschiemer

• Staging Database: deployment/stacks/postgresql-staging/

  • Container: postgres-staging
  • Network: postgres-staging-internal
  • Database: michaelschiemer_staging

Migration Steps

Phase 1: Backup Existing Database

CRITICAL: Create backups before migration!

# On production server
cd ~/deployment/stacks/postgresql

# Create backup of production database
docker exec postgres-backup /scripts/backup.sh

# Create backup of staging database (if exists in same instance)
docker exec postgres psql -U postgres -d michaelschiemer_staging -c "SELECT 1;" && \
  docker exec postgres-backup /scripts/backup.sh || echo "Staging database not found"

# List backups
ls -lh backups/

Phase 2: Deploy New Database Stacks

2.1 Deploy PostgreSQL Production Stack

# On production server
cd ~/deployment/stacks/postgresql-production

# Create .env file
cat > .env <<EOF
POSTGRES_DB=michaelschiemer
POSTGRES_USER=postgres
POSTGRES_PASSWORD=<same-as-production-password>
BACKUP_RETENTION_DAYS=7
BACKUP_SCHEDULE=0 2 * * *
EOF

# Start PostgreSQL Production
docker compose up -d

# Verify
docker compose ps
docker exec postgres-production pg_isready -U postgres -d michaelschiemer

2.2 Deploy PostgreSQL Staging Stack

# On production server
cd ~/deployment/stacks/postgresql-staging

# Create .env file
cat > .env <<EOF
POSTGRES_DB=michaelschiemer_staging
POSTGRES_USER=postgres
POSTGRES_PASSWORD=<staging-password-can-differ>
BACKUP_RETENTION_DAYS=3
BACKUP_SCHEDULE=0 3 * * *
EOF

# Start PostgreSQL Staging
docker compose up -d

# Verify
docker compose ps
docker exec postgres-staging pg_isready -U postgres -d michaelschiemer_staging

Phase 3: Restore Data to New Stacks

3.1 Restore Production Database

# Find latest production backup
LATEST_PROD_BACKUP=$(ls -t ~/deployment/stacks/postgresql/backups/postgres_michaelschiemer_*.sql.gz | head -1)

# Copy backup to new stack
cp "$LATEST_PROD_BACKUP" ~/deployment/stacks/postgresql-production/backups/

# Restore to new production database
cd ~/deployment/stacks/postgresql-production
docker exec -it postgres-production-backup /scripts/restore.sh "/backups/$(basename $LATEST_PROD_BACKUP)"

3.2 Restore Staging Database (if exists)

# Find latest staging backup (if exists)
LATEST_STAGING_BACKUP=$(ls -t ~/deployment/stacks/postgresql/backups/postgres_michaelschiemer_staging_*.sql.gz 2>/dev/null | head -1)

if [ -n "$LATEST_STAGING_BACKUP" ]; then
  # Copy backup to new stack
  cp "$LATEST_STAGING_BACKUP" ~/deployment/stacks/postgresql-staging/backups/
  
  # Restore to new staging database
  cd ~/deployment/stacks/postgresql-staging
  docker exec -it postgres-staging-backup /scripts/restore.sh "/backups/$(basename $LATEST_STAGING_BACKUP)"
else
  echo "No staging backup found - creating empty staging database"
fi

Phase 4: Update Application Stacks

4.1 Update Production Application Stack

The Production Application Stack should already be configured to use postgres-production via:

• Environment variable: DB_HOST=postgres-production
• Network: Connected to postgres-production-internal

Verify connection:

cd ~/deployment/stacks/production
docker compose -f docker-compose.base.yml -f docker-compose.production.yml exec app \
  php -r "echo 'DB Connection: '; \$pdo = new PDO('pgsql:host=postgres-production;dbname=michaelschiemer', 'postgres', getenv('DB_PASSWORD')); echo 'OK\n';"

4.2 Update Staging Application Stack

The Staging Application Stack should already be configured to use postgres-staging via:

• Environment variable: DB_HOST=postgres-staging
• Network: Connected to postgres-staging-internal

Verify connection:

cd ~/deployment/stacks/staging
docker compose -f docker-compose.base.yml -f docker-compose.staging.yml exec staging-app \
  php -r "echo 'DB Connection: '; \$pdo = new PDO('pgsql:host=postgres-staging;dbname=michaelschiemer_staging', 'postgres', getenv('DB_PASSWORD')); echo 'OK\n';"

Phase 5: Verify Migration

5.1 Production Verification

# Check production database
docker exec postgres-production psql -U postgres -d michaelschiemer -c "\dt"

# Check production application
curl -k https://michaelschiemer.de/health

# Check production logs
cd ~/deployment/stacks/production
docker compose logs app | tail -20

5.2 Staging Verification

# Check staging database
docker exec postgres-staging psql -U postgres -d michaelschiemer_staging -c "\dt"

# Check staging application
curl -k https://staging.michaelschiemer.de/health

# Check staging logs
cd ~/deployment/stacks/staging
docker compose logs staging-app | tail -20

Phase 6: Cleanup (Optional)

WARNING: Only remove old stack after successful migration and verification!

# Stop old PostgreSQL stack
cd ~/deployment/stacks/postgresql
docker compose down

# Keep backups for safety - do NOT delete yet
# Consider keeping for at least 30 days

# Remove old stack (only after 30+ days of successful operation)
# rm -rf ~/deployment/stacks/postgresql

Rollback Plan

If migration fails:

Rollback to Shared Database

# 1. Stop new database stacks
cd ~/deployment/stacks/postgresql-production
docker compose down

cd ~/deployment/stacks/postgresql-staging
docker compose down

# 2. Restart old shared stack
cd ~/deployment/stacks/postgresql
docker compose up -d

# 3. Update application stacks to use old database
# Revert DB_HOST changes in docker-compose files
# Restart application stacks

Network Configuration

Production Application Stack

Must be connected to:

• traefik-public (for external access)
• postgres-production-internal (for database access)
• app-internal (for Redis, if shared)

Staging Application Stack

Must be connected to:

• traefik-public (for external access)
• postgres-staging-internal (for database access)
• staging-internal (for staging services)

Backup Strategy

Production Backups

• Location: ~/deployment/stacks/postgresql-production/backups/
• Retention: 7 days
• Schedule: Daily at 2:00 AM

Staging Backups

• Location: ~/deployment/stacks/postgresql-staging/backups/
• Retention: 3 days
• Schedule: Daily at 3:00 AM

Troubleshooting

Database Connection Refused

# Check if database container is running
docker ps | grep postgres-production
docker ps | grep postgres-staging

# Check network connectivity
docker network inspect postgres-production-internal
docker network inspect postgres-staging-internal

# Test connection from application container
docker exec <app-container> nc -zv postgres-production 5432
docker exec <app-container> nc -zv postgres-staging 5432

Migration Data Loss

If data is missing after migration:

1. Stop application to prevent further writes
2. Restore from backup (see Phase 3)
3. Verify data before restarting application

Additional Resources

• PostgreSQL Production Stack: deployment/stacks/postgresql-production/README.md
• PostgreSQL Staging Stack: deployment/stacks/postgresql-staging/README.md
• Backup Playbook: deployment/ansible/playbooks/backup.yml