michaelschiemer/deployment/docs/guides/migrate-to-separate-databases.md

# Migration Guide: Separate Database Stacks

## Overview

This guide describes how to migrate from a shared PostgreSQL stack to separate database stacks for Production and Staging.

## Current State (Before Migration)

- Single PostgreSQL stack: `deployment/stacks/postgresql/`
- Both Production and Staging connect to the same database instance
- Database name: `michaelschiemer` (Production) and `michaelschiemer_staging` (Staging) in same instance

## Target State (After Migration)

- **Production Database**: `deployment/stacks/postgresql-production/`
  - Container: `postgres-production`
  - Network: `postgres-production-internal`
  - Database: `michaelschiemer`

- **Staging Database**: `deployment/stacks/postgresql-staging/`
  - Container: `postgres-staging`
  - Network: `postgres-staging-internal`
  - Database: `michaelschiemer_staging`

## Migration Steps

### Phase 1: Backup Existing Database

**CRITICAL**: Create backups before migration!

```bash
# On production server
cd ~/deployment/stacks/postgresql

# Create backup of production database
docker exec postgres-backup /scripts/backup.sh

# Create backup of staging database (if exists in same instance)
docker exec postgres psql -U postgres -d michaelschiemer_staging -c "SELECT 1;" && \
  docker exec postgres-backup /scripts/backup.sh || echo "Staging database not found"

# List backups
ls -lh backups/
```

### Phase 2: Deploy New Database Stacks

#### 2.1 Deploy PostgreSQL Production Stack

```bash
# On production server
cd ~/deployment/stacks/postgresql-production

# Create .env file
cat > .env <<EOF
POSTGRES_DB=michaelschiemer
POSTGRES_USER=postgres
POSTGRES_PASSWORD=<same-as-production-password>
BACKUP_RETENTION_DAYS=7
BACKUP_SCHEDULE=0 2 * * *
EOF

# Start PostgreSQL Production
docker compose up -d

# Verify
docker compose ps
docker exec postgres-production pg_isready -U postgres -d michaelschiemer
```

#### 2.2 Deploy PostgreSQL Staging Stack

```bash
# On production server
cd ~/deployment/stacks/postgresql-staging

# Create .env file
cat > .env <<EOF
POSTGRES_DB=michaelschiemer_staging
POSTGRES_USER=postgres
POSTGRES_PASSWORD=<staging-password-can-differ>
BACKUP_RETENTION_DAYS=3
BACKUP_SCHEDULE=0 3 * * *
EOF

# Start PostgreSQL Staging
docker compose up -d

# Verify
docker compose ps
docker exec postgres-staging pg_isready -U postgres -d michaelschiemer_staging
```

### Phase 3: Restore Data to New Stacks

#### 3.1 Restore Production Database

```bash
# Find latest production backup
LATEST_PROD_BACKUP=$(ls -t ~/deployment/stacks/postgresql/backups/postgres_michaelschiemer_*.sql.gz | head -1)

# Copy backup to new stack
cp "$LATEST_PROD_BACKUP" ~/deployment/stacks/postgresql-production/backups/

# Restore to new production database
cd ~/deployment/stacks/postgresql-production
docker exec -it postgres-production-backup /scripts/restore.sh "/backups/$(basename $LATEST_PROD_BACKUP)"
```

#### 3.2 Restore Staging Database (if exists)

```bash
# Find latest staging backup (if exists)
LATEST_STAGING_BACKUP=$(ls -t ~/deployment/stacks/postgresql/backups/postgres_michaelschiemer_staging_*.sql.gz 2>/dev/null | head -1)

if [ -n "$LATEST_STAGING_BACKUP" ]; then
  # Copy backup to new stack
  cp "$LATEST_STAGING_BACKUP" ~/deployment/stacks/postgresql-staging/backups/
  
  # Restore to new staging database
  cd ~/deployment/stacks/postgresql-staging
  docker exec -it postgres-staging-backup /scripts/restore.sh "/backups/$(basename $LATEST_STAGING_BACKUP)"
else
  echo "No staging backup found - creating empty staging database"
fi
```

### Phase 4: Update Application Stacks

#### 4.1 Update Production Application Stack

The Production Application Stack should already be configured to use `postgres-production` via:
- Environment variable: `DB_HOST=postgres-production`
- Network: Connected to `postgres-production-internal`

**Verify connection**:
```bash
cd ~/deployment/stacks/production
docker compose -f docker-compose.base.yml -f docker-compose.production.yml exec app \
  php -r "echo 'DB Connection: '; \$pdo = new PDO('pgsql:host=postgres-production;dbname=michaelschiemer', 'postgres', getenv('DB_PASSWORD')); echo 'OK\n';"
```

#### 4.2 Update Staging Application Stack

The Staging Application Stack should already be configured to use `postgres-staging` via:
- Environment variable: `DB_HOST=postgres-staging`
- Network: Connected to `postgres-staging-internal`

**Verify connection**:
```bash
cd ~/deployment/stacks/staging
docker compose -f docker-compose.base.yml -f docker-compose.staging.yml exec staging-app \
  php -r "echo 'DB Connection: '; \$pdo = new PDO('pgsql:host=postgres-staging;dbname=michaelschiemer_staging', 'postgres', getenv('DB_PASSWORD')); echo 'OK\n';"
```

### Phase 5: Verify Migration

#### 5.1 Production Verification

```bash
# Check production database
docker exec postgres-production psql -U postgres -d michaelschiemer -c "\dt"

# Check production application
curl -k https://michaelschiemer.de/health

# Check production logs
cd ~/deployment/stacks/production
docker compose logs app | tail -20
```

#### 5.2 Staging Verification

```bash
# Check staging database
docker exec postgres-staging psql -U postgres -d michaelschiemer_staging -c "\dt"

# Check staging application
curl -k https://staging.michaelschiemer.de/health

# Check staging logs
cd ~/deployment/stacks/staging
docker compose logs staging-app | tail -20
```

### Phase 6: Cleanup (Optional)

**WARNING**: Only remove old stack after successful migration and verification!

```bash
# Stop old PostgreSQL stack
cd ~/deployment/stacks/postgresql
docker compose down

# Keep backups for safety - do NOT delete yet
# Consider keeping for at least 30 days

# Remove old stack (only after 30+ days of successful operation)
# rm -rf ~/deployment/stacks/postgresql
```

## Rollback Plan

If migration fails:

### Rollback to Shared Database

```bash
# 1. Stop new database stacks
cd ~/deployment/stacks/postgresql-production
docker compose down

cd ~/deployment/stacks/postgresql-staging
docker compose down

# 2. Restart old shared stack
cd ~/deployment/stacks/postgresql
docker compose up -d

# 3. Update application stacks to use old database
# Revert DB_HOST changes in docker-compose files
# Restart application stacks
```

## Network Configuration

### Production Application Stack

Must be connected to:
- `traefik-public` (for external access)
- `postgres-production-internal` (for database access)
- `app-internal` (for Redis, if shared)

### Staging Application Stack

Must be connected to:
- `traefik-public` (for external access)
- `postgres-staging-internal` (for database access)
- `staging-internal` (for staging services)

## Backup Strategy

### Production Backups

- **Location**: `~/deployment/stacks/postgresql-production/backups/`
- **Retention**: 7 days
- **Schedule**: Daily at 2:00 AM

### Staging Backups

- **Location**: `~/deployment/stacks/postgresql-staging/backups/`
- **Retention**: 3 days
- **Schedule**: Daily at 3:00 AM

## Troubleshooting

### Database Connection Refused

```bash
# Check if database container is running
docker ps | grep postgres-production
docker ps | grep postgres-staging

# Check network connectivity
docker network inspect postgres-production-internal
docker network inspect postgres-staging-internal

# Test connection from application container
docker exec <app-container> nc -zv postgres-production 5432
docker exec <app-container> nc -zv postgres-staging 5432
```

### Migration Data Loss

If data is missing after migration:

1. **Stop application** to prevent further writes
2. **Restore from backup** (see Phase 3)
3. **Verify data** before restarting application

## Additional Resources

- **PostgreSQL Production Stack**: `deployment/stacks/postgresql-production/README.md`
- **PostgreSQL Staging Stack**: `deployment/stacks/postgresql-staging/README.md`
- **Backup Playbook**: `deployment/ansible/playbooks/backup.yml`