michaelschiemer/docs/logging/best-practices.md

# Logging Best Practices

Comprehensive guide for effective logging in production with the Custom PHP Framework.

## Overview

This document provides best practices for implementing, configuring, and maintaining the logging infrastructure in production environments.

## Choosing the Right Handler

### RotatingFileHandler vs Standard FileHandler

**Use RotatingFileHandler when**:
- Logs grow continuously and need automatic management
- Disk space constraints require automatic cleanup
- Historical log retention with automatic archival is needed
- Production environments requiring hands-off log management

**Use Standard FileHandler when**:
- External log rotation tools (logrotate) are already in place
- Logs are shipped to external systems immediately
- Development/testing environments with manual cleanup
- Very low-volume logging scenarios

### Factory Method Selection

```php
use App\Framework\Logging\Handlers\RotatingFileHandler;
use App\Framework\Core\ValueObjects\Byte;

// Production: INFO and above, daily rotation, 30 days retention
$handler = RotatingFileHandler::production('/var/www/html/storage/logs/app.log');

// Daily rotation: New log file every day at midnight
$handler = RotatingFileHandler::daily(
    logFile: '/var/www/html/storage/logs/app.log',
    maxFiles: 30,  // Keep 30 days of logs
    compress: true  // Compress rotated files (saves ~70% disk space)
);

// Weekly rotation: New log file every Monday
$handler = RotatingFileHandler::weekly(
    logFile: '/var/www/html/storage/logs/weekly.log',
    maxFiles: 12,  // Keep 3 months of weekly logs
    compress: true
);

// Size-based rotation: Rotate when file reaches size limit
$handler = RotatingFileHandler::withSizeRotation(
    logFile: '/var/www/html/storage/logs/api.log',
    maxFileSize: Byte::fromMegabytes(100),  // Rotate at 100MB
    maxFiles: 10,  // Keep 10 rotated files
    compress: true
);
```

## Rotation Strategy Selection

### Daily Rotation

**Best for**:
- Application logs with moderate volume (1-100 MB/day)
- Compliance requirements (daily log archival)
- Debugging workflows that align with daily releases/deployments
- General production application logging

**Configuration**:
```php
$handler = RotatingFileHandler::daily(
    logFile: '/var/www/html/storage/logs/app.log',
    maxFiles: 30,      // 30 days retention
    compress: true,    // Save disk space
    minLevel: LogLevel::INFO  // Production: INFO and above
);
```

**Retention Guidelines**:
- Development: 7 days
- Staging: 14 days
- Production: 30-90 days (compliance dependent)

### Weekly Rotation

**Best for**:
- Low-volume logs (<10 MB/day)
- Background job logs
- Scheduled task logs
- Non-critical system logs

**Configuration**:
```php
$handler = RotatingFileHandler::weekly(
    logFile: '/var/www/html/storage/logs/background.log',
    maxFiles: 12,      // 3 months retention
    compress: true
);
```

### Size-Based Rotation

**Best for**:
- High-volume API logs
- Variable traffic patterns (burst handling)
- Logs where size control is more important than time boundaries
- Debug logs during development

**Configuration**:
```php
// High-traffic API logs
$handler = RotatingFileHandler::withSizeRotation(
    logFile: '/var/www/html/storage/logs/api.log',
    maxFileSize: Byte::fromMegabytes(100),
    maxFiles: 20,  // 2GB total (20 × 100MB)
    compress: true
);

// Development: Smaller files for easier viewing
$handler = RotatingFileHandler::withSizeRotation(
    logFile: '/var/www/html/storage/logs/debug.log',
    maxFileSize: Byte::fromMegabytes(10),
    maxFiles: 5,
    compress: false  // No compression in dev for easier access
);
```

**Size Recommendations**:
- Development: 10-50 MB
- Staging: 50-100 MB
- Production: 100-500 MB (balance between rotation frequency and file count)

## Log Level Guidelines

### Production Log Levels

```php
use App\Framework\Logging\LogLevel;

// Production: INFO and above (default)
$handler = RotatingFileHandler::production($logFile);

// Custom production configuration
$handler = RotatingFileHandler::daily($logFile)
    ->withMinLevel(LogLevel::INFO);
```

**Level Selection**:
- **ERROR**: Critical failures requiring immediate attention
- **WARNING**: Potentially harmful situations that recovered
- **INFO**: Important business events (user registration, order completion)
- **DEBUG**: Detailed development information (disabled in production)

**Production Recommendations**:
- Main application log: INFO and above
- Security log: WARNING and above
- API access log: INFO (all requests)
- Background jobs: INFO (job start/completion)

### Development Log Levels

```php
// Development: All levels including DEBUG
$handler = RotatingFileHandler::daily(
    logFile: '/var/www/html/storage/logs/dev.log',
    maxFiles: 7,
    compress: false
)->withMinLevel(LogLevel::DEBUG);
```

### Separate Log Files by Severity

```php
// Error-only log for critical monitoring
$errorHandler = RotatingFileHandler::daily(
    logFile: '/var/www/html/storage/logs/errors.log',
    maxFiles: 90  // Keep errors longer
)->withMinLevel(LogLevel::ERROR);

// General application log
$appHandler = RotatingFileHandler::production(
    '/var/www/html/storage/logs/app.log'
);

// Attach both handlers to logger
$logger->addHandler($errorHandler);
$logger->addHandler($appHandler);
```

## Production Configuration Examples

### Standard Web Application

```php
// Application logs: Daily rotation, 30 days retention
$appHandler = RotatingFileHandler::production(
    '/var/www/html/storage/logs/app.log'
);

// Error logs: Keep longer for pattern analysis
$errorHandler = RotatingFileHandler::daily(
    logFile: '/var/www/html/storage/logs/errors.log',
    maxFiles: 90,
    compress: true
)->withMinLevel(LogLevel::ERROR);

// Security logs: Weekly rotation, compressed
$securityHandler = RotatingFileHandler::weekly(
    logFile: '/var/www/html/storage/logs/security.log',
    maxFiles: 52,  // 1 year
    compress: true
);
```

### High-Traffic API

```php
// API access logs: Size-based rotation for burst traffic
$apiHandler = RotatingFileHandler::withSizeRotation(
    logFile: '/var/www/html/storage/logs/api.log',
    maxFileSize: Byte::fromMegabytes(200),
    maxFiles: 20,  // 4GB total
    compress: true
)->withMinLevel(LogLevel::INFO);

// API errors: Separate file for monitoring
$apiErrorHandler = RotatingFileHandler::daily(
    logFile: '/var/www/html/storage/logs/api-errors.log',
    maxFiles: 60
)->withMinLevel(LogLevel::ERROR);
```

### Background Job Processing

```php
// Job logs: Daily rotation
$jobHandler = RotatingFileHandler::daily(
    logFile: '/var/www/html/storage/logs/jobs.log',
    maxFiles: 14,  // 2 weeks sufficient for job logs
    compress: true
)->withMinLevel(LogLevel::INFO);

// Failed job logs: Keep longer for debugging
$failedJobHandler = RotatingFileHandler::weekly(
    logFile: '/var/www/html/storage/logs/failed-jobs.log',
    maxFiles: 12,  // 3 months
    compress: true
)->withMinLevel(LogLevel::ERROR);
```

## Compression Best Practices

### When to Enable Compression

**Enable compression when**:
- Disk space is limited
- Log retention period is >7 days
- Logs are rarely accessed after rotation
- Production environments

**Disable compression when**:
- Development/debugging (frequent log access)
- Logs are shipped to external systems immediately
- Performance is critical (compression adds overhead)
- Very low log volume (<10 MB/day)

### Compression Trade-offs

```php
// Production: Compression enabled (recommended)
$handler = RotatingFileHandler::production($logFile);

// Development: Compression disabled for easier access
$handler = RotatingFileHandler::daily(
    logFile: $logFile,
    maxFiles: 7,
    compress: false  // No compression in dev
);
```

**Benefits**:
- Space savings: ~70% reduction on average
- Reduced backup costs
- Longer retention within disk limits

**Costs**:
- CPU overhead during rotation (negligible)
- Requires decompression to read old logs
- Slightly slower rotation process

## Health Monitoring

### LogHealthCheckCommand Usage

```bash
# Manual health check
docker exec php php console.php logs:health-check

# Detailed output with file information
docker exec php php console.php logs:health-check --detailed

# Auto-fix permission issues
docker exec php php console.php logs:health-check --fix-permissions
```

### Scheduled Health Checks

```php
use App\Framework\Scheduler\Services\SchedulerService;
use App\Framework\Scheduler\Schedules\CronSchedule;

// Daily health check at 2 AM
$scheduler->schedule(
    'log-health-check',
    CronSchedule::fromExpression('0 2 * * *'),
    function() {
        $command = new LogHealthCheckCommand();
        $input = new ConsoleInput(['logs:health-check'], new ConsoleOutput(), null);
        $exitCode = $command->execute($input);

        if ($exitCode !== ExitCode::SUCCESS) {
            // Alert operations team
            $this->alerting->sendAlert(
                level: AlertLevel::WARNING,
                message: 'Log health check failed'
            );
        }

        return ['exit_code' => $exitCode];
    }
);
```

### Health Check Metrics

Monitor these metrics from health checks:
- **Directory existence**: All required log directories present
- **Write permissions**: Can write to log directories
- **Disk space**: >100MB free (warning), >10% free (healthy)
- **Large files**: Individual files >50MB (rotation issue)
- **Total log size**: Track growth trends

### Automated Alerts

```php
// Alert on health check failures
final readonly class LogHealthMonitor
{
    public function checkAndAlert(): void
    {
        $health = $this->runHealthCheck();

        if (!$health['directories_ok']) {
            $this->alerting->sendAlert(
                level: AlertLevel::CRITICAL,
                message: 'Log directories missing or inaccessible',
                context: $health['directory_issues']
            );
        }

        if ($health['disk_space_low']) {
            $this->alerting->sendAlert(
                level: AlertLevel::WARNING,
                message: "Low disk space: {$health['free_space']}",
                context: $health['disk_info']
            );
        }

        if ($health['large_files_count'] > 0) {
            $this->alerting->sendAlert(
                level: AlertLevel::INFO,
                message: "{$health['large_files_count']} log files exceed 50MB",
                context: $health['large_files']
            );
        }
    }
}
```

## Docker/Container Deployment

### Directory Structure

```
storage/
└── logs/
    ├── app/            # Application logs
    ├── debug/          # Debug logs (dev only)
    ├── security/       # Security events
    └── jobs/           # Background job logs
```

### Docker Volume Configuration

```yaml
# docker-compose.yml
services:
  php:
    volumes:
      # Host mount for logs (accessible from host for monitoring)
      - ./storage/logs:/var/www/html/storage/logs
```

**Benefits of host mounts**:
- Logs persist across container restarts
- Direct access from host for monitoring tools
- Backup integration without entering container
- Log rotation tools on host can access files

### Permissions Setup

```bash
# Fix permissions for Docker container
chmod -R 777 storage/logs

# Or use Alpine container for one-time fix
docker run --rm -v $(pwd)/storage:/mnt alpine chmod -R 777 /mnt/logs
```

### Production Container Configuration

```dockerfile
# Dockerfile
FROM php:8.3-fpm

# Create log directories
RUN mkdir -p /var/www/html/storage/logs/app && \
    mkdir -p /var/www/html/storage/logs/security && \
    mkdir -p /var/www/html/storage/logs/jobs && \
    chown -R www-data:www-data /var/www/html/storage && \
    chmod -R 755 /var/www/html/storage/logs
```

## Performance Considerations

### Log Volume Management

**High-volume scenarios**:
```php
// Use size-based rotation for predictable disk usage
$handler = RotatingFileHandler::withSizeRotation(
    logFile: '/var/www/html/storage/logs/high-volume.log',
    maxFileSize: Byte::fromMegabytes(100),
    maxFiles: 20,
    compress: true
);

// Separate handlers for different log types
$errorHandler = RotatingFileHandler::daily($errorLog)
    ->withMinLevel(LogLevel::ERROR);  // Errors only

$debugHandler = RotatingFileHandler::withSizeRotation($debugLog, Byte::fromMegabytes(50))
    ->withMinLevel(LogLevel::DEBUG);  // Debug info
```

**Performance Benchmarks** (from integration tests):
- 1000 log entries: <5 seconds
- Rotation overhead: <100ms per rotation
- Compression overhead: ~200ms for 100MB file

### Memory Management

```php
// For very high-volume logging, consider async logging
use App\Framework\Queue\Queue;

final readonly class AsyncLoggingHandler implements LogHandler
{
    public function handle(LogRecord $record): void
    {
        // Queue log writes for background processing
        $job = new WriteLogJob($record);
        $this->queue->push(JobPayload::immediate($job));
    }
}
```

### Disk I/O Optimization

**Best practices**:
- Write logs synchronously for errors (data loss prevention)
- Buffer INFO/DEBUG logs for batch writes (performance)
- Use separate handlers for different log types (parallel writes)
- Monitor disk I/O metrics during rotation

## Security Considerations

### Log File Permissions

```bash
# Recommended permissions
chmod 755 storage/logs          # Directory: Read + execute for all, write for owner
chmod 644 storage/logs/*.log    # Files: Read for all, write for owner
```

**Security guidelines**:
- Never 777 permissions in production (security risk)
- Use 755 for directories (allows reading but prevents unauthorized writes)
- Use 644 for files (readable by monitoring tools, writable only by application)
- Separate user for web server and log access

### Sensitive Data Logging

**DO NOT log**:
- Passwords (plaintext or hashed)
- API keys and secrets
- Credit card numbers
- Personal identification numbers (SSN, etc.)
- Session tokens

**Safe logging patterns**:
```php
// ❌ Unsafe: Logs sensitive data
$logger->info('User login', [
    'password' => $password,  // NEVER log passwords
    'api_key' => $apiKey      // NEVER log keys
]);

// ✅ Safe: Logs only necessary information
$logger->info('User login', [
    'user_id' => $userId,
    'ip_address' => $ipAddress,
    'user_agent' => $userAgent
]);

// ✅ Safe: Mask sensitive data
$logger->info('Payment processed', [
    'card_last_four' => substr($cardNumber, -4),  // Only last 4 digits
    'amount' => $amount,
    'transaction_id' => $transactionId
]);
```

### Security Event Logging

```php
// Use framework's OWASP security logger for security events
use App\Framework\Security\OWASPSecurityLogger;
use App\Framework\Security\OWASPEventIdentifier;

$this->owaspLogger->logSecurityEvent(
    new SecurityEventType(OWASPEventIdentifier::AUTHN_LOGIN_FAILURE),
    request: $request,
    context: [
        'username' => $username,
        'ip_address' => $request->server->getRemoteAddr(),
        'failure_reason' => 'Invalid credentials'
    ]
);
```

## Troubleshooting

### Common Issues

#### Issue: Logs not rotating

**Symptoms**:
- Log file grows continuously
- No `.1`, `.2` rotation files created

**Diagnosis**:
```bash
# Check file permissions
ls -la storage/logs/

# Check disk space
df -h

# Verify handler configuration
docker exec php php console.php logs:health-check --detailed
```

**Solutions**:
```php
// Verify rotation configuration
$handler = RotatingFileHandler::daily($logFile, maxFiles: 30);

// Check if logs are actually being written
if (!file_exists($logFile)) {
    // Permission issue or incorrect path
}

// Manually trigger rotation for testing
touch($logFile, strtotime('yesterday'));
$handler->handle($logRecord);  // Should rotate now
```

#### Issue: Permission denied errors

**Symptoms**:
- Cannot write to log files
- Health check reports permission errors

**Solutions**:
```bash
# Fix permissions (development)
chmod -R 777 storage/logs

# Fix permissions (production - more secure)
chown -R www-data:www-data storage/logs
chmod -R 755 storage/logs
chmod -R 644 storage/logs/*.log

# Use health check auto-fix
docker exec php php console.php logs:health-check --fix-permissions
```

#### Issue: Disk space full

**Symptoms**:
- Health check reports low disk space
- Log rotation fails

**Solutions**:
```bash
# Check disk usage
du -sh storage/logs/*

# Find largest log files
find storage/logs -type f -exec ls -lh {} \; | sort -k5 -rh | head -10

# Manually clean old logs
find storage/logs -name "*.log.*" -mtime +30 -delete

# Increase compression for existing logs
find storage/logs -name "*.log.[0-9]*" ! -name "*.gz" -exec gzip {} \;
```

**Prevention**:
```php
// Reduce retention period
$handler = RotatingFileHandler::daily($logFile, maxFiles: 14);  // 14 days instead of 30

// Reduce file size for size-based rotation
$handler = RotatingFileHandler::withSizeRotation(
    $logFile,
    maxFileSize: Byte::fromMegabytes(50),  // 50MB instead of 100MB
    maxFiles: 10
);

// Enable compression if not already
$handler = RotatingFileHandler::daily($logFile, compress: true);
```

#### Issue: Performance degradation

**Symptoms**:
- Slow request processing
- High disk I/O during logging

**Solutions**:
```php
// Separate handlers by log type to parallelize writes
$errorHandler = RotatingFileHandler::daily('/var/www/html/storage/logs/errors.log')
    ->withMinLevel(LogLevel::ERROR);

$infoHandler = RotatingFileHandler::daily('/var/www/html/storage/logs/info.log')
    ->withMinLevel(LogLevel::INFO)
    ->withMaxLevel(LogLevel::WARNING);

// Consider async logging for high-volume scenarios
use App\Framework\Logging\Handlers\QueuedLogHandler;

$asyncHandler = new QueuedLogHandler($queue, $handler);
```

## Integration with Framework Systems

### Scheduler Integration

```php
// Automated log rotation cleanup
$scheduler->schedule(
    'log-cleanup',
    CronSchedule::fromExpression('0 3 * * *'),  // Daily at 3 AM
    function() {
        // Remove logs older than retention policy
        $this->logCleanupService->cleanupExpiredLogs();

        return ['cleaned' => true];
    }
);

// Health monitoring
$scheduler->schedule(
    'log-health-check',
    IntervalSchedule::every(Duration::fromHours(6)),
    function() {
        $command = new LogHealthCheckCommand();
        $input = new ConsoleInput(['logs:health-check'], new ConsoleOutput(), null);

        return ['exit_code' => $command->execute($input)];
    }
);
```

### Queue Integration

```php
// Background log processing for high-volume scenarios
final class ProcessLogBatchJob
{
    public function handle(): void
    {
        $logBatch = $this->logBuffer->flush();

        foreach ($logBatch as $logRecord) {
            $this->handler->handle($logRecord);
        }
    }
}
```

### Event System Integration

```php
// Log important domain events
use App\Framework\Event\EventHandler;

#[EventHandler]
final readonly class UserRegistrationLogger
{
    public function handle(UserRegisteredEvent $event): void
    {
        $this->logger->info('User registered', [
            'user_id' => $event->userId,
            'email' => $event->email->getMasked(),
            'registration_method' => $event->method
        ]);
    }
}
```

## Monitoring and Alerting

### Metrics to Track

1. **Log Volume Metrics**:
   - Logs per hour/day
   - Growth rate trends
   - Rotation frequency

2. **Performance Metrics**:
   - Write latency (should be <10ms)
   - Rotation duration (should be <1s)
   - Disk I/O utilization

3. **Health Metrics**:
   - Disk space free percentage
   - Number of files exceeding size thresholds
   - Permission check failures

4. **Error Metrics**:
   - Error log volume (sudden spikes indicate issues)
   - Failed rotation attempts
   - Write permission errors

### Alert Configuration

```php
// Example monitoring integration
final readonly class LogMonitoringService
{
    public function collectMetrics(): array
    {
        return [
            'total_log_size' => $this->getTotalLogSize(),
            'log_file_count' => $this->getLogFileCount(),
            'largest_file_size' => $this->getLargestFileSize(),
            'disk_space_free' => $this->getDiskSpaceFree(),
            'rotation_count_24h' => $this->getRotationCount(Duration::fromHours(24))
        ];
    }

    public function checkThresholds(array $metrics): array
    {
        $alerts = [];

        if ($metrics['disk_space_free'] < 1_000_000_000) {  // <1GB
            $alerts[] = [
                'level' => 'critical',
                'message' => 'Log disk space critically low'
            ];
        }

        if ($metrics['largest_file_size'] > 500_000_000) {  // >500MB
            $alerts[] = [
                'level' => 'warning',
                'message' => 'Large log file detected (rotation issue?)'
            ];
        }

        return $alerts;
    }
}
```

## Summary Checklist

### Production Deployment

- [ ] Configure appropriate rotation strategy (daily/weekly/size-based)
- [ ] Set retention period based on compliance requirements
- [ ] Enable compression for rotated files
- [ ] Set production log level (INFO and above)
- [ ] Configure separate log files by severity (errors, security, etc.)
- [ ] Set up Docker volume mounts for log persistence
- [ ] Fix file permissions (755 directories, 644 files)
- [ ] Schedule health checks (daily recommended)
- [ ] Configure monitoring and alerting
- [ ] Test log rotation under load
- [ ] Document log access procedures for operations team

### Development Setup

- [ ] Use DEBUG log level for development
- [ ] Disable compression for easier access
- [ ] Use shorter retention (7 days sufficient)
- [ ] Smaller file size limits for size-based rotation
- [ ] Regular cleanup to prevent disk space issues
- [ ] Test rotation locally before deploying

### Ongoing Maintenance

- [ ] Monitor disk space weekly
- [ ] Review health check results daily
- [ ] Analyze log volume trends monthly
- [ ] Clean up old logs exceeding retention period
- [ ] Review and update alerting thresholds quarterly
- [ ] Test disaster recovery procedures (log restoration)
- [ ] Update documentation as logging patterns evolve

## Additional Resources

- **LogHealthCheckCommand**: `docs/logging/log-health-check-command.md`
- **RotatingFileHandler Implementation**: `src/Framework/Logging/Handlers/RotatingFileHandler.php`
- **Integration Tests**: `tests/Integration/Framework/Logging/LoggingInfrastructureTest.php`
- **OWASP Security Logging**: `docs/claude/security-patterns.md`