michaelschiemer/deployment/docs/DEPLOYMENT_GUIDE.md

# SSH Deployment Guide

Comprehensive guide for deploying the Custom PHP Framework using SSH-based deployment scripts.

## Overview

This deployment system uses simple SSH/SCP-based scripts to deploy the framework to staging and production environments. It replaces Gitea Actions with a straightforward bash script approach.

**Key Features**:
- ✅ Simple SSH/SCP deployment (no CI/CD platform dependency)
- ✅ Automatic Docker image building and registry pushing
- ✅ Database backups before production deployments
- ✅ Automatic rollback on deployment failure
- ✅ Health checks and smoke tests
- ✅ Timestamped backup retention
- ✅ Color-coded output for easy monitoring

## Prerequisites

### Required Software

**Local Machine**:
- Docker (for building images)
- Docker Compose (for compose file validation)
- SSH client (openssh-client)
- SCP client (usually bundled with SSH)
- Bash shell

**Remote Servers** (staging/production):
- Docker and Docker Compose installed
- SSH server running
- Docker private registry accessible (localhost:5000 or custom)
- Deployment user with Docker permissions
- Directory structure: `/opt/framework-staging/` or `/opt/framework-production/`

### SSH Key Setup

Generate SSH keys for deployment (if not already done):

```bash
# Generate deployment SSH key
ssh-keygen -t rsa -b 4096 -f ~/.ssh/framework-deploy \
  -C "framework-deployment" -N ""

# Copy public key to staging server
ssh-copy-id -i ~/.ssh/framework-deploy.pub deploy@staging.michaelschiemer.de

# Copy public key to production server
ssh-copy-id -i ~/.ssh/framework-deploy.pub deploy@michaelschiemer.de

# Test connection
ssh -i ~/.ssh/framework-deploy deploy@staging.michaelschiemer.de "echo 'SSH connection successful'"
```

**SSH Config** (~/.ssh/config):
```
# Staging Server
Host staging.michaelschiemer.de
    User deploy
    IdentityFile ~/.ssh/framework-deploy
    Port 22

# Production Server
Host michaelschiemer.de
    User deploy
    IdentityFile ~/.ssh/framework-deploy
    Port 22
```

### Environment Variables

**Staging Deployment**:
```bash
export STAGING_HOST=staging.michaelschiemer.de
export STAGING_USER=deploy
export STAGING_SSH_PORT=22
```

**Production Deployment**:
```bash
export PRODUCTION_HOST=michaelschiemer.de
export PRODUCTION_USER=deploy
export PRODUCTION_SSH_PORT=22
```

**Optional Configuration**:
```bash
# Docker Registry (default: localhost:5000)
export REGISTRY=your-registry.com

# Image Configuration
export IMAGE_NAME=framework
export IMAGE_TAG=latest  # or staging

# Production Options
export SKIP_BACKUP=false  # Skip database backup (not recommended)
export FORCE_REBUILD=false  # Force Docker rebuild
```

**Persistent Configuration** (.bashrc or .zshrc):
```bash
# Add to ~/.bashrc or ~/.zshrc
export STAGING_HOST=staging.michaelschiemer.de
export STAGING_USER=deploy
export PRODUCTION_HOST=michaelschiemer.de
export PRODUCTION_USER=deploy
```

## Deployment Scripts

### 1. Staging Deployment

**Script**: `deployment/scripts/deploy-staging.sh`

**Purpose**: Deploy to staging environment for testing

**Usage**:
```bash
# Basic deployment
./deployment/scripts/deploy-staging.sh

# With custom configuration
STAGING_HOST=custom.staging.com ./deployment/scripts/deploy-staging.sh
```

**What It Does**:
1. Builds Docker image with `ENV=staging`
2. Pushes image to private registry
3. Creates timestamped backup of current deployment
4. Copies deployment files via SCP
5. Stops existing containers
6. Starts new containers
7. Waits 30 seconds for services to initialize
8. Performs health checks
9. Automatic rollback on failure

**Backup Retention**: Keeps last 5 backups, deletes older

**Deployment Path**: `/opt/framework-staging/current/`

**Expected Output**:
```
==================================================
🚀 Starting Staging Deployment
==================================================
Registry: localhost:5000
Image: framework:staging
Remote: deploy@staging.michaelschiemer.de:22
Path: /opt/framework-staging

[1/7] Building Docker image...
[2/7] Pushing image to registry...
[3/7] Preparing deployment files...
[4/7] Creating remote directory and backup...
Backing up current deployment...
Backup created: backup_20250124_153022
[5/7] Copying deployment files to server...
[6/7] Executing deployment on server...
==================================================
Starting Staging Deployment on Server
==================================================
[1/5] Pulling latest Docker images...
[2/5] Stopping existing containers...
[3/5] Starting new containers...
[4/5] Waiting for services to be healthy...
[5/5] Verifying deployment...
==================================================
✅ Staging Deployment Complete
==================================================
[7/7] Performing health checks...
Waiting 30 seconds for services to initialize...
Checking container status...
✅ Health check complete!

==================================================
✅ Staging Deployment Successful
==================================================
URL: https://staging.michaelschiemer.de
Deployed at: Thu Jan 24 15:30:45 CET 2025
```

### 2. Production Deployment

**Script**: `deployment/scripts/deploy-production.sh`

**Purpose**: Deploy to production environment

**⚠️ WARNING**: Production deployments include:
- Automatic database backup (mandatory unless skipped)
- 60-second service initialization wait
- Smoke tests for main page and API health
- Automatic rollback on any failure

**Usage**:
```bash
# Standard production deployment
./deployment/scripts/deploy-production.sh

# Skip database backup (NOT RECOMMENDED)
SKIP_BACKUP=true ./deployment/scripts/deploy-production.sh

# Force Docker rebuild
FORCE_REBUILD=true ./deployment/scripts/deploy-production.sh
```

**What It Does**:
1. Builds Docker image with `ENV=production`
2. Pushes image to private registry
3. **Creates database backup** (aborts if backup fails)
4. Creates timestamped backup of current deployment
5. Copies deployment files via SCP
6. Stops existing containers gracefully
7. Starts new containers
8. Waits 60 seconds for services to initialize
9. Runs database migrations with `--force`
10. Performs comprehensive health checks:
    - Container status
    - PHP-FPM process check
    - Redis connection test
11. **Runs smoke tests**:
    - Main page accessibility (https://michaelschiemer.de/)
    - API health endpoint (https://michaelschiemer.de/api/health)
12. Automatic rollback on any failure

**Backup Retention**: Keeps last 10 backups, deletes older

**Deployment Path**: `/opt/framework-production/current/`

**Database Backup Location**: `/var/www/html/storage/backups/backup_YYYYMMDD_HHMMSS.sql`

**Expected Output**:
```
==================================================
🚀 Starting Production Deployment
==================================================
Registry: localhost:5000
Image: framework:latest
Remote: deploy@michaelschiemer.de:22
Path: /opt/framework-production
Skip Backup: false

[1/8] Building Docker image...
[2/8] Pushing image to registry...
[3/8] Preparing deployment files...
[4/8] Creating remote directory and backup...
[5/8] Copying deployment files to server...
[6/8] Executing deployment on server...
==================================================
Starting Production Deployment on Server
==================================================
[0/6] Creating database backup...
✅ Database backup created: backup_20250124_153045.sql
[1/6] Pulling latest Docker images...
[2/6] Stopping existing containers (graceful shutdown)...
[3/6] Starting new containers...
[4/6] Waiting for services to be healthy...
[5/6] Running database migrations...
[6/6] Verifying deployment...
==================================================
✅ Production Deployment Complete
==================================================
[7/8] Performing health checks...
Waiting 60 seconds for services to initialize...
Checking container status...
✅ All health checks passed!
[8/8] Running smoke tests...
✅ Main page accessible
✅ API health check passed
✅ Smoke tests completed successfully

==================================================
✅ Production Deployment Successful
==================================================
URL: https://michaelschiemer.de
Deployed at: Thu Jan 24 15:32:15 CET 2025
```

### 3. Rollback Script

**Script**: `deployment/scripts/rollback.sh`

**Purpose**: Restore previous deployment from backup

**Usage**:
```bash
# Rollback staging to latest backup
./deployment/scripts/rollback.sh staging

# Rollback production to latest backup
./deployment/scripts/rollback.sh production

# Rollback to specific backup
./deployment/scripts/rollback.sh production backup_20250124_143022
```

**What It Does**:
1. Lists available backups
2. Confirms rollback operation (requires "yes")
3. Stops current deployment
4. Archives failed deployment as `failed_YYYYMMDD_HHMMSS`
5. Restores specified backup
6. Starts restored deployment
7. Performs health checks

**Arguments**:
- `environment`: `staging` or `production` (required)
- `backup_name`: Specific backup to restore (optional, defaults to latest)

**Example Session**:
```bash
$ ./deployment/scripts/rollback.sh production

==================================================
🔄 Starting Rollback: production
==================================================
Remote: deploy@michaelschiemer.de:22
Path: /opt/framework-production
Target Backup: Latest available

⚠️  WARNING: This will rollback the production deployment
   Current deployment will be stopped and replaced with backup

Are you sure you want to continue? (yes/no): yes

[1/5] Listing available backups...
Available backups:
backup_20250124_153045
backup_20250124_120000
backup_20250123_183015

[2/5] Determining backup to restore...
Using latest backup: backup_20250124_153045
✅ Backup backup_20250124_153045 verified

[3/5] Stopping current deployment...
✅ Current deployment stopped

[4/5] Restoring backup...
Archiving failed deployment as failed_20250124_154512...
Restoring backup backup_20250124_153045...
✅ Backup restored

[5/5] Starting restored deployment...
Starting containers...
Waiting for services to start...
✅ Restored deployment is running

==================================================
✅ Rollback Complete
==================================================
Environment: production
Restored: backup_20250124_153045
Completed at: Thu Jan 24 15:45:30 CET 2025

Failed deployment archived as: failed_20250124_154512
```

## Deployment Workflows

### Staging Deployment Workflow

**Step-by-Step Process**:

1. **Prepare Changes**:
   ```bash
   # Make code changes locally
   git add .
   git commit -m "feat: new feature"
   git push origin staging
   ```

2. **Deploy to Staging**:
   ```bash
   # Set environment variables (if not in ~/.bashrc)
   export STAGING_HOST=staging.michaelschiemer.de
   export STAGING_USER=deploy

   # Run deployment
   ./deployment/scripts/deploy-staging.sh
   ```

3. **Verify Deployment**:
   ```bash
   # Check application
   curl -k https://staging.michaelschiemer.de/health

   # Monitor logs
   ssh deploy@staging.michaelschiemer.de \
     "cd /opt/framework-staging/current && docker-compose logs -f"

   # Check container status
   ssh deploy@staging.michaelschiemer.de \
     "cd /opt/framework-staging/current && docker-compose ps"
   ```

4. **Test Application**:
   - Perform manual testing
   - Run automated tests
   - Verify feature functionality
   - Check performance

5. **If Issues Found**:
   ```bash
   # Rollback staging
   ./deployment/scripts/rollback.sh staging

   # Or continue testing for non-critical issues
   ```

### Production Deployment Workflow

**Step-by-Step Process**:

1. **Pre-Deployment Checklist**:
   - [ ] Code reviewed and approved
   - [ ] Successfully deployed and tested in staging
   - [ ] Database migrations tested
   - [ ] Backup plan confirmed
   - [ ] Rollback plan confirmed
   - [ ] Team notified of deployment window

2. **Prepare Production Branch**:
   ```bash
   # Merge staging to main
   git checkout main
   git merge staging
   git push origin main
   ```

3. **Verify Environment Variables**:
   ```bash
   # Required variables
   echo $PRODUCTION_HOST  # Should be: michaelschiemer.de
   echo $PRODUCTION_USER  # Should be: deploy

   # If not set
   export PRODUCTION_HOST=michaelschiemer.de
   export PRODUCTION_USER=deploy
   ```

4. **Deploy to Production**:
   ```bash
   # IMPORTANT: Do NOT skip database backup
   ./deployment/scripts/deploy-production.sh

   # Monitor output carefully for any errors
   ```

5. **Post-Deployment Verification**:
   ```bash
   # 1. Check main application
   curl -k https://michaelschiemer.de/

   # 2. Check API health
   curl -k https://michaelschiemer.de/api/health

   # 3. Monitor logs for errors
   ssh deploy@michaelschiemer.de \
     "cd /opt/framework-production/current && docker-compose logs -f --tail=100"

   # 4. Check container status
   ssh deploy@michaelschiemer.de \
     "cd /opt/framework-production/current && docker-compose ps"

   # 5. Verify database migrations applied
   ssh deploy@michaelschiemer.de \
     "cd /opt/framework-production/current && \
      docker-compose exec production-app php console.php db:status"
   ```

6. **Smoke Testing**:
   - Test critical user paths
   - Verify authentication
   - Test key API endpoints
   - Check database connectivity
   - Verify external integrations

7. **If Deployment Fails**:
   ```bash
   # Automatic rollback should have occurred
   # If manual rollback needed:
   ./deployment/scripts/rollback.sh production

   # Monitor rollback
   ssh deploy@michaelschiemer.de \
     "cd /opt/framework-production/current && docker-compose logs -f"
   ```

8. **Post-Deployment**:
   - Monitor application metrics
   - Watch error logs for 30 minutes
   - Notify team of successful deployment
   - Document any issues encountered

## Troubleshooting

### SSH Connection Issues

**Problem**: `Permission denied (publickey)`

**Solutions**:
```bash
# Verify SSH key exists
ls -la ~/.ssh/framework-deploy*

# Test SSH connection
ssh -i ~/.ssh/framework-deploy deploy@staging.michaelschiemer.de "echo 'SSH works'"

# Check SSH config
cat ~/.ssh/config

# Re-copy public key
ssh-copy-id -i ~/.ssh/framework-deploy.pub deploy@staging.michaelschiemer.de

# Check server-side authorized_keys
ssh deploy@staging.michaelschiemer.de "cat ~/.ssh/authorized_keys"
```

### Docker Build Failures

**Problem**: Docker build fails during deployment

**Solutions**:
```bash
# Check Docker is running
docker info

# Test build locally
docker build \
  --file docker/php/Dockerfile \
  --tag localhost:5000/framework:test \
  --build-arg ENV=staging \
  .

# Check Dockerfile syntax
docker build --file docker/php/Dockerfile --no-cache .

# Clear Docker cache
docker system prune -a
```

### Registry Push Failures

**Problem**: `docker push` fails

**Solutions**:
```bash
# Check registry is accessible
curl http://localhost:5000/v2/

# Verify image exists locally
docker images | grep framework

# Test manual push
docker push localhost:5000/framework:staging

# Check registry logs
docker logs registry  # If running registry as container
```

### Deployment Script Fails

**Problem**: Deployment script exits with error

**Solutions**:
```bash
# Run with bash debug mode
bash -x ./deployment/scripts/deploy-staging.sh

# Check remote directory exists
ssh deploy@staging.michaelschiemer.de "ls -la /opt/framework-staging"

# Verify Docker Compose files
ssh deploy@staging.michaelschiemer.de \
  "cd /opt/framework-staging/current && docker-compose config"

# Check deployment logs on server
ssh deploy@staging.michaelschiemer.de \
  "cd /opt/framework-staging/current && docker-compose logs"
```

### Health Check Failures

**Problem**: Health checks fail but containers are running

**Solutions**:
```bash
# Check container logs
ssh deploy@staging.michaelschiemer.de \
  "cd /opt/framework-staging/current && docker-compose logs --tail=50"

# Check PHP-FPM status
ssh deploy@staging.michaelschiemer.de \
  "cd /opt/framework-staging/current && \
   docker-compose exec staging-app pgrep php-fpm"

# Test health endpoint manually
ssh deploy@staging.michaelschiemer.de \
  "curl -k http://localhost/health"

# Check Nginx configuration
ssh deploy@staging.michaelschiemer.de \
  "cd /opt/framework-staging/current && \
   docker-compose exec staging-nginx nginx -t"
```

### Rollback Issues

**Problem**: Rollback script fails

**Solutions**:
```bash
# List available backups
ssh deploy@production \
  "cd /opt/framework-production && ls -dt backup_*"

# Manually restore backup
ssh deploy@production "
  cd /opt/framework-production
  docker-compose -f current/docker-compose.base.yml \
                 -f current/docker-compose.prod.yml down
  rm -rf current
  cp -r backup_20250124_153045 current
  cd current
  docker-compose -f docker-compose.base.yml \
                 -f docker-compose.prod.yml up -d
"

# Check failed deployment archive
ssh deploy@production "ls -dt /opt/framework-production/failed_*"
```

### Database Migration Failures

**Problem**: Migrations fail during deployment

**Solutions**:
```bash
# Check migration status
ssh deploy@production \
  "cd /opt/framework-production/current && \
   docker-compose exec production-app php console.php db:status"

# Manually run migrations
ssh deploy@production \
  "cd /opt/framework-production/current && \
   docker-compose exec production-app php console.php db:migrate --force"

# Rollback migrations
ssh deploy@production \
  "cd /opt/framework-production/current && \
   docker-compose exec production-app php console.php db:rollback"

# Check database connectivity
ssh deploy@production \
  "cd /opt/framework-production/current && \
   docker-compose exec production-app php console.php db:check"
```

## Security Best Practices

### SSH Key Management

**✅ Do**:
- Use 4096-bit RSA keys minimum
- Generate separate keys for staging and production
- Store private keys securely (never commit to git)
- Rotate keys quarterly
- Use SSH config for key management

**❌ Don't**:
- Use password-only authentication
- Share keys between environments
- Commit private keys to version control
- Use personal SSH keys for deployments

### Environment Variables

**✅ Do**:
- Use environment variables for secrets
- Document required variables
- Use different credentials per environment
- Validate variables before deployment

**❌ Don't**:
- Hard-code credentials in scripts
- Commit .env files with secrets
- Use production credentials in staging

### Deployment User Permissions

**Recommended Setup**:
```bash
# On remote server
# Create deployment user
sudo useradd -m -s /bin/bash deploy

# Add to docker group
sudo usermod -aG docker deploy

# Set directory ownership
sudo chown -R deploy:deploy /opt/framework-staging
sudo chown -R deploy:deploy /opt/framework-production

# Restrict sudo (if needed)
# Add to /etc/sudoers.d/deploy
deploy ALL=(ALL) NOPASSWD: /usr/bin/docker, /usr/bin/docker-compose
```

### Backup Management

**✅ Do**:
- Automate database backups
- Keep multiple backup versions
- Test backup restoration regularly
- Monitor backup disk space

**❌ Don't**:
- Skip backups in production
- Keep unlimited backups (disk space)
- Store backups only on deployment server

## Monitoring and Maintenance

### Health Monitoring

**Automated Checks**:
```bash
# Cron job for health monitoring
# Add to crontab -e on deployment server
*/5 * * * * curl -f -k https://michaelschiemer.de/health || echo "Health check failed" | mail -s "Production Health Alert" admin@michaelschiemer.de
```

**Manual Checks**:
```bash
# Check all services
ssh deploy@production \
  "cd /opt/framework-production/current && docker-compose ps"

# Check resource usage
ssh deploy@production "docker stats --no-stream"

# Check disk space
ssh deploy@production "df -h /opt/framework-production"
```

### Log Management

**View Logs**:
```bash
# Follow logs
ssh deploy@production \
  "cd /opt/framework-production/current && docker-compose logs -f"

# View specific service logs
ssh deploy@production \
  "cd /opt/framework-production/current && \
   docker-compose logs -f production-app"

# Last 100 lines
ssh deploy@production \
  "cd /opt/framework-production/current && \
   docker-compose logs --tail=100"
```

### Backup Cleanup

**Manual Cleanup**:
```bash
# List backups by size
ssh deploy@production "du -sh /opt/framework-production/backup_* | sort -h"

# Remove specific old backup
ssh deploy@production "rm -rf /opt/framework-production/backup_20240101_000000"

# Keep only last 5 backups
ssh deploy@staging "
  cd /opt/framework-staging
  ls -dt backup_* | tail -n +6 | xargs rm -rf
"
```

## Appendix

### Directory Structure

**Local Project**:
```
michaelschiemer/
├── deployment/
│   ├── scripts/
│   │   ├── deploy-staging.sh       # Staging deployment
│   │   ├── deploy-production.sh    # Production deployment
│   │   └── rollback.sh             # Rollback script
│   ├── docs/
│   │   └── DEPLOYMENT_GUIDE.md     # This file
│   └── legacy/
│       └── gitea-workflows/        # Archived Gitea workflows
├── docker-compose.base.yml
├── docker-compose.staging.yml
├── docker-compose.prod.yml
└── docker/
    └── php/
        └── Dockerfile
```

**Remote Server**:
```
/opt/framework-staging/    or    /opt/framework-production/
├── current/                          # Active deployment
│   ├── docker-compose.base.yml
│   ├── docker-compose.staging.yml
│   ├── docker/
│   └── deploy.sh
├── backup_20250124_153045/          # Timestamped backups
├── backup_20250124_120000/
├── backup_20250123_183015/
└── failed_20250124_154512/          # Failed deployment (if rollback occurred)
```

### Environment Variable Reference

| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `STAGING_HOST` | Yes* | staging.michaelschiemer.de | Staging server hostname/IP |
| `STAGING_USER` | No | deploy | Staging SSH user |
| `STAGING_SSH_PORT` | No | 22 | Staging SSH port |
| `PRODUCTION_HOST` | Yes* | michaelschiemer.de | Production server hostname/IP |
| `PRODUCTION_USER` | No | deploy | Production SSH user |
| `PRODUCTION_SSH_PORT` | No | 22 | Production SSH port |
| `REGISTRY` | No | localhost:5000 | Docker registry URL |
| `IMAGE_NAME` | No | framework | Docker image name |
| `IMAGE_TAG` | No | staging/latest | Docker image tag |
| `SKIP_BACKUP` | No | false | Skip database backup (production) |
| `FORCE_REBUILD` | No | false | Force Docker image rebuild |

*Required for respective deployment type

### Common Commands Reference

**Local Commands**:
```bash
# Deploy staging
./deployment/scripts/deploy-staging.sh

# Deploy production
./deployment/scripts/deploy-production.sh

# Rollback staging
./deployment/scripts/rollback.sh staging

# Rollback production
./deployment/scripts/rollback.sh production

# Test SSH connection
ssh deploy@staging.michaelschiemer.de "echo 'SSH works'"
```

**Remote Commands** (via SSH):
```bash
# View logs
docker-compose logs -f

# Check status
docker-compose ps

# Restart services
docker-compose restart

# Stop services
docker-compose down

# Start services
docker-compose up -d

# Execute command in container
docker-compose exec production-app php console.php db:status

# View container logs
docker-compose logs production-app --tail=50
```

---

**Last Updated**: 2025-01-24
**Framework Version**: 2.x
**Deployment Method**: SSH-based deployment scripts