michaelschiemer/deployment/docs/TROUBLESHOOTING.md

# Troubleshooting Guide

This guide helps you diagnose and fix common deployment issues for the Custom PHP Framework.

## Project Information
- **Domain**: michaelschiemer.de
- **Email**: kontakt@michaelschiemer.de  
- **PHP Version**: 8.4

## Quick Diagnostics

### System Status Check

```bash
# Check overall deployment status
make status ENV=staging

# Run health checks
make health ENV=production

# Check prerequisites
make check-prerequisites

# Validate configuration
make validate-config ENV=production
```

### Log Investigation

```bash
# View application logs
make logs ENV=staging

# Infrastructure logs
tail -f deployment/infrastructure/logs/ansible.log

# Docker container logs
docker-compose logs --tail=100 -f php
docker-compose logs --tail=100 -f nginx
docker-compose logs --tail=100 -f db
```

## Common Issues and Solutions

### 1. Setup and Prerequisites

#### Issue: Dependencies Not Installed

**Symptoms:**
```bash
command not found: docker
command not found: ansible-playbook
```

**Solution:**
```bash
# Run setup script
./setup.sh

# Or install manually
sudo apt-get install docker.io docker-compose ansible  # Ubuntu/Debian
brew install docker ansible  # macOS
```

#### Issue: Docker Permission Denied

**Symptoms:**
```bash
Got permission denied while trying to connect to the Docker daemon socket
```

**Solution:**
```bash
# Add user to docker group
sudo usermod -aG docker $USER

# Log out and back in, or start new shell
newgrp docker

# Test Docker access
docker ps
```

#### Issue: SSH Key Authentication

**Symptoms:**
```bash
Permission denied (publickey)
```

**Solution:**
```bash
# Generate SSH keys if not exists
ssh-keygen -t ed25519 -C "deployment@michaelschiemer.de"

# Add public key to target server
ssh-copy-id user@your-server.com

# Or manually copy key
cat ~/.ssh/id_ed25519.pub
# Copy output to server's ~/.ssh/authorized_keys
```

### 2. Configuration Issues

#### Issue: Environment File Not Found

**Symptoms:**
```bash
ERROR: Environment file not found: .env.production
```

**Solution:**
```bash
# Create from template
cp applications/environments/.env.production.template applications/environments/.env.production

# Or initialize all configs
make init-config

# Edit the configuration
make edit-config ENV=production
```

#### Issue: Template Values Not Replaced

**Symptoms:**
```bash
ERROR: Environment file contains unfilled templates
```

**Solution:**
```bash
# Find unfilled templates
grep "*** REQUIRED" applications/environments/.env.production

# Replace with actual values
nano applications/environments/.env.production

# Generate secure passwords
openssl rand -base64 32 | tr -d "=+/" | cut -c1-25
```

#### Issue: SSL Certificate Problems

**Symptoms:**
```bash
SSL certificate error
nginx: [emerg] cannot load certificate
```

**Solutions:**
```bash
# For Let's Encrypt issues
# Check domain DNS points to server
dig +short michaelschiemer.de

# Verify SSL email is correct
grep SSL_EMAIL applications/environments/.env.production

# For self-signed certificates (development)
# Regenerate certificates
./scripts/generate_ssl_certificates.sh

# Check certificate validity
openssl x509 -in /path/to/cert.pem -text -noout
```

### 3. Deployment Failures

#### Issue: Ansible Connection Failed

**Symptoms:**
```bash
UNREACHABLE! => {"msg": "Failed to connect to the host via ssh"}
```

**Solutions:**
```bash
# Test SSH connection manually
ssh user@your-server.com

# Check Ansible inventory
cat deployment/infrastructure/inventories/production/hosts.yml

# Test Ansible connectivity
ansible all -i deployment/infrastructure/inventories/production/hosts.yml -m ping

# Common fixes:
# 1. Update server IP address in inventory
# 2. Ensure SSH key is added to server
# 3. Check firewall allows SSH (port 22)
# 4. Verify username in inventory file
```

#### Issue: Docker Compose Build Failed

**Symptoms:**
```bash
ERROR: Failed to build custom-php-framework
```

**Solutions:**
```bash
# Check Docker Compose syntax
docker-compose config

# Rebuild without cache
docker-compose build --no-cache

# Check for disk space
df -h

# Clear Docker build cache
docker system prune -a

# Check specific service logs
docker-compose logs php
```

#### Issue: Database Connection Failed

**Symptoms:**
```bash
SQLSTATE[HY000] [2002] Connection refused
```

**Solutions:**
```bash
# Check database container status
docker-compose ps db

# Check database logs
docker-compose logs db

# Test database connection
docker-compose exec php php console.php db:ping

# Verify database credentials in .env file
grep DB_ applications/environments/.env.production

# Reset database container
docker-compose down
docker volume rm michaelschiemer_db_data  # WARNING: This removes all data
docker-compose up -d db
```

### 4. Application Issues

#### Issue: 502 Bad Gateway

**Symptoms:**
- Nginx shows 502 error
- Application not responding

**Solutions:**
```bash
# Check if PHP-FPM container is running
docker-compose ps php

# Check PHP-FPM logs
docker-compose logs php

# Restart PHP container
docker-compose restart php

# Check nginx upstream configuration
docker-compose exec nginx nginx -t

# Verify PHP-FPM is listening on correct port
docker-compose exec php netstat -ln | grep 9000
```

#### Issue: 404 Not Found

**Symptoms:**
- All routes return 404
- Static files not found

**Solutions:**
```bash
# Check nginx configuration
docker-compose exec nginx nginx -t

# Check document root
docker-compose exec php ls -la /var/www/html/public/

# Verify file permissions
docker-compose exec php chmod -R 755 /var/www/html/public
docker-compose exec php chown -R www-data:www-data /var/www/html

# Check nginx routing
docker-compose logs nginx | grep 404
```

#### Issue: PHP Fatal Errors

**Symptoms:**
```bash
PHP Fatal error: Class not found
```

**Solutions:**
```bash
# Check composer autoloader
docker-compose exec php composer dump-autoload -o

# Verify dependencies installed
docker-compose exec php composer install --no-dev --optimize-autoloader

# Check PHP configuration
docker-compose exec php php -i | grep -E "(memory_limit|max_execution_time)"

# Check application logs
docker-compose logs php | grep "FATAL"
```

### 5. Performance Issues

#### Issue: Slow Response Times

**Symptoms:**
- Pages load slowly
- Timeouts occur

**Solutions:**
```bash
# Check resource usage
docker stats

# Monitor PHP-FPM processes
docker-compose exec php ps aux | grep php-fpm

# Check database queries
docker-compose logs db | grep "Query_time"

# Optimize PHP-FPM configuration
# Edit: deployment/applications/dockerfiles/php-fpm/php-fpm.conf

# Enable OPcache
docker-compose exec php php -m | grep OPcache
```

#### Issue: High Memory Usage

**Symptoms:**
```bash
Fatal error: Allowed memory size exhausted
```

**Solutions:**
```bash
# Increase PHP memory limit
# Edit .env file: PHP_MEMORY_LIMIT=1024M

# Check memory usage
docker-compose exec php php -r "echo ini_get('memory_limit');"

# Monitor memory usage
docker stats --no-stream

# Restart containers with new limits
docker-compose down && docker-compose up -d
```

### 6. SSL and Security Issues

#### Issue: SSL Certificate Not Trusted

**Symptoms:**
- Browser shows security warning
- SSL certificate invalid

**Solutions:**
```bash
# Check certificate status
curl -I https://michaelschiemer.de

# Verify certificate chain
openssl s_client -connect michaelschiemer.de:443 -servername michaelschiemer.de

# For Let's Encrypt renewal issues
docker-compose exec nginx certbot renew --dry-run

# Check certificate expiration
echo | openssl s_client -connect michaelschiemer.de:443 2>/dev/null | openssl x509 -noout -dates
```

#### Issue: Firewall Blocking Connections

**Symptoms:**
- Connection timeout
- Cannot reach server

**Solutions:**
```bash
# Check firewall status on server
sudo ufw status

# Allow HTTP/HTTPS traffic
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 22/tcp  # SSH

# Check if ports are listening
netstat -tlnp | grep :80
netstat -tlnp | grep :443
```

## Advanced Troubleshooting

### Debug Mode

Enable debug mode for detailed error information:

```bash
# Enable debug in .env file (non-production only)
APP_DEBUG=true

# Redeploy with debug enabled
./deploy.sh staging --force

# Check detailed logs
make logs ENV=staging | grep ERROR
```

### Verbose Deployment

Run deployment with verbose output:

```bash
# Verbose deployment
./deploy.sh staging --verbose

# Dry run with verbose output
./deploy.sh production --dry-run --verbose

# Ansible verbose mode
cd deployment/infrastructure
ansible-playbook -i inventories/staging/hosts.yml site.yml -vvv
```

### Database Debugging

```bash
# Check database status
make health ENV=staging

# Access database directly
docker-compose exec db mysql -u root -p

# Check database structure
docker-compose exec php php console.php db:status

# Run migrations
docker-compose exec php php console.php db:migrate

# Rollback migrations
docker-compose exec php php console.php db:rollback
```

### Container Debugging

```bash
# Enter container for debugging
docker-compose exec php /bin/bash
docker-compose exec nginx /bin/sh

# Check container resource usage
docker stats --no-stream

# Inspect container configuration
docker-compose config

# Check container networking
docker network ls
docker network inspect michaelschiemer_default
```

## Recovery Procedures

### Emergency Procedures

```bash
# Emergency stop all services
make emergency-stop ENV=production

# Emergency restart
make emergency-restart ENV=production

# Rollback deployment (with caution)
make rollback ENV=production
```

### Backup and Restore

```bash
# Create backup before troubleshooting
make backup ENV=production

# Restore from backup if needed
make restore ENV=production

# List available backups
ls -la ../storage/backups/
```

### Service Recovery

```bash
# Restart specific service
docker-compose restart nginx
docker-compose restart php
docker-compose restart db

# Rebuild and restart
docker-compose down
docker-compose up -d --build

# Full reset (removes data - use with caution)
docker-compose down -v
docker-compose up -d
```

## Getting Help

### Check Documentation

1. Review [Quick Start Guide](QUICKSTART.md)
2. Check [Environment Configuration](ENVIRONMENTS.md)
3. Examine deployment logs

### Collect Information

Before asking for help, collect:
- Error messages from logs
- Environment configuration (sanitized)
- System information (`docker --version`, `ansible --version`)
- Deployment command used
- Environment being deployed to

### Common Commands for Support

```bash
# System information
make version
make info

# Configuration status
make validate-config ENV=production

# Health checks
make health ENV=staging

# Recent logs
make logs ENV=production | tail -100
```

### Emergency Contacts

For critical production issues:
- Check system logs immediately
- Create backup if possible
- Document the issue and steps taken
- Consider rollback if service is down

## Prevention

### Best Practices

1. **Always test with dry-run first**
   ```bash
   ./deploy.sh production --dry-run
   ```

2. **Use staging environment**
   ```bash
   make deploy-staging
   # Test thoroughly before production
   ```

3. **Regular backups**
   ```bash
   make backup ENV=production
   ```

4. **Monitor health**
   ```bash
   make health ENV=production
   ```

5. **Keep configuration secure**
   ```bash
   chmod 600 applications/environments/.env.*
   ```

### Monitoring Setup

Consider implementing:
- Automated health checks
- Log monitoring and alerting
- Performance monitoring
- SSL certificate expiration alerts
- Database backup verification

This troubleshooting guide should help you resolve most common deployment issues. Remember to always test changes in a staging environment before applying them to production!