michaelschiemer/docs/logging/best-practices.md

Logging Best Practices

Comprehensive guide for effective logging in production with the Custom PHP Framework.

Overview

This document provides best practices for implementing, configuring, and maintaining the logging infrastructure in production environments.

Choosing the Right Handler

RotatingFileHandler vs Standard FileHandler

Use RotatingFileHandler when:

• Logs grow continuously and need automatic management
• Disk space constraints require automatic cleanup
• Historical log retention with automatic archival is needed
• Production environments requiring hands-off log management

Use Standard FileHandler when:

• External log rotation tools (logrotate) are already in place
• Logs are shipped to external systems immediately
• Development/testing environments with manual cleanup
• Very low-volume logging scenarios

Factory Method Selection

use App\Framework\Logging\Handlers\RotatingFileHandler;
use App\Framework\Core\ValueObjects\Byte;

// Production: INFO and above, daily rotation, 30 days retention
$handler = RotatingFileHandler::production('/var/www/html/storage/logs/app.log');

// Daily rotation: New log file every day at midnight
$handler = RotatingFileHandler::daily(
    logFile: '/var/www/html/storage/logs/app.log',
    maxFiles: 30,  // Keep 30 days of logs
    compress: true  // Compress rotated files (saves ~70% disk space)
);

// Weekly rotation: New log file every Monday
$handler = RotatingFileHandler::weekly(
    logFile: '/var/www/html/storage/logs/weekly.log',
    maxFiles: 12,  // Keep 3 months of weekly logs
    compress: true
);

// Size-based rotation: Rotate when file reaches size limit
$handler = RotatingFileHandler::withSizeRotation(
    logFile: '/var/www/html/storage/logs/api.log',
    maxFileSize: Byte::fromMegabytes(100),  // Rotate at 100MB
    maxFiles: 10,  // Keep 10 rotated files
    compress: true
);

Rotation Strategy Selection

Daily Rotation

Best for:

• Application logs with moderate volume (1-100 MB/day)
• Compliance requirements (daily log archival)
• Debugging workflows that align with daily releases/deployments
• General production application logging

Configuration:

$handler = RotatingFileHandler::daily(
    logFile: '/var/www/html/storage/logs/app.log',
    maxFiles: 30,      // 30 days retention
    compress: true,    // Save disk space
    minLevel: LogLevel::INFO  // Production: INFO and above
);

Retention Guidelines:

• Development: 7 days
• Staging: 14 days
• Production: 30-90 days (compliance dependent)

Weekly Rotation

Best for:

• Low-volume logs (<10 MB/day)
• Background job logs
• Scheduled task logs
• Non-critical system logs

Configuration:

$handler = RotatingFileHandler::weekly(
    logFile: '/var/www/html/storage/logs/background.log',
    maxFiles: 12,      // 3 months retention
    compress: true
);

Size-Based Rotation

Best for:

• High-volume API logs
• Variable traffic patterns (burst handling)
• Logs where size control is more important than time boundaries
• Debug logs during development

Configuration:

// High-traffic API logs
$handler = RotatingFileHandler::withSizeRotation(
    logFile: '/var/www/html/storage/logs/api.log',
    maxFileSize: Byte::fromMegabytes(100),
    maxFiles: 20,  // 2GB total (20 × 100MB)
    compress: true
);

// Development: Smaller files for easier viewing
$handler = RotatingFileHandler::withSizeRotation(
    logFile: '/var/www/html/storage/logs/debug.log',
    maxFileSize: Byte::fromMegabytes(10),
    maxFiles: 5,
    compress: false  // No compression in dev for easier access
);

Size Recommendations:

• Development: 10-50 MB
• Staging: 50-100 MB
• Production: 100-500 MB (balance between rotation frequency and file count)

Log Level Guidelines

Production Log Levels

use App\Framework\Logging\LogLevel;

// Production: INFO and above (default)
$handler = RotatingFileHandler::production($logFile);

// Custom production configuration
$handler = RotatingFileHandler::daily($logFile)
    ->withMinLevel(LogLevel::INFO);

Level Selection:

• ERROR: Critical failures requiring immediate attention
• WARNING: Potentially harmful situations that recovered
• INFO: Important business events (user registration, order completion)
• DEBUG: Detailed development information (disabled in production)

Production Recommendations:

• Main application log: INFO and above
• Security log: WARNING and above
• API access log: INFO (all requests)
• Background jobs: INFO (job start/completion)

Development Log Levels

// Development: All levels including DEBUG
$handler = RotatingFileHandler::daily(
    logFile: '/var/www/html/storage/logs/dev.log',
    maxFiles: 7,
    compress: false
)->withMinLevel(LogLevel::DEBUG);

Separate Log Files by Severity

// Error-only log for critical monitoring
$errorHandler = RotatingFileHandler::daily(
    logFile: '/var/www/html/storage/logs/errors.log',
    maxFiles: 90  // Keep errors longer
)->withMinLevel(LogLevel::ERROR);

// General application log
$appHandler = RotatingFileHandler::production(
    '/var/www/html/storage/logs/app.log'
);

// Attach both handlers to logger
$logger->addHandler($errorHandler);
$logger->addHandler($appHandler);

Production Configuration Examples

Standard Web Application

// Application logs: Daily rotation, 30 days retention
$appHandler = RotatingFileHandler::production(
    '/var/www/html/storage/logs/app.log'
);

// Error logs: Keep longer for pattern analysis
$errorHandler = RotatingFileHandler::daily(
    logFile: '/var/www/html/storage/logs/errors.log',
    maxFiles: 90,
    compress: true
)->withMinLevel(LogLevel::ERROR);

// Security logs: Weekly rotation, compressed
$securityHandler = RotatingFileHandler::weekly(
    logFile: '/var/www/html/storage/logs/security.log',
    maxFiles: 52,  // 1 year
    compress: true
);

High-Traffic API

// API access logs: Size-based rotation for burst traffic
$apiHandler = RotatingFileHandler::withSizeRotation(
    logFile: '/var/www/html/storage/logs/api.log',
    maxFileSize: Byte::fromMegabytes(200),
    maxFiles: 20,  // 4GB total
    compress: true
)->withMinLevel(LogLevel::INFO);

// API errors: Separate file for monitoring
$apiErrorHandler = RotatingFileHandler::daily(
    logFile: '/var/www/html/storage/logs/api-errors.log',
    maxFiles: 60
)->withMinLevel(LogLevel::ERROR);

Background Job Processing

// Job logs: Daily rotation
$jobHandler = RotatingFileHandler::daily(
    logFile: '/var/www/html/storage/logs/jobs.log',
    maxFiles: 14,  // 2 weeks sufficient for job logs
    compress: true
)->withMinLevel(LogLevel::INFO);

// Failed job logs: Keep longer for debugging
$failedJobHandler = RotatingFileHandler::weekly(
    logFile: '/var/www/html/storage/logs/failed-jobs.log',
    maxFiles: 12,  // 3 months
    compress: true
)->withMinLevel(LogLevel::ERROR);

Compression Best Practices

When to Enable Compression

Enable compression when:

• Disk space is limited
• Log retention period is >7 days
• Logs are rarely accessed after rotation
• Production environments

Disable compression when:

• Development/debugging (frequent log access)
• Logs are shipped to external systems immediately
• Performance is critical (compression adds overhead)
• Very low log volume (<10 MB/day)

Compression Trade-offs

// Production: Compression enabled (recommended)
$handler = RotatingFileHandler::production($logFile);

// Development: Compression disabled for easier access
$handler = RotatingFileHandler::daily(
    logFile: $logFile,
    maxFiles: 7,
    compress: false  // No compression in dev
);

Benefits:

• Space savings: ~70% reduction on average
• Reduced backup costs
• Longer retention within disk limits

Costs:

• CPU overhead during rotation (negligible)
• Requires decompression to read old logs
• Slightly slower rotation process

Health Monitoring

LogHealthCheckCommand Usage

# Manual health check
docker exec php php console.php logs:health-check

# Detailed output with file information
docker exec php php console.php logs:health-check --detailed

# Auto-fix permission issues
docker exec php php console.php logs:health-check --fix-permissions

Scheduled Health Checks

use App\Framework\Scheduler\Services\SchedulerService;
use App\Framework\Scheduler\Schedules\CronSchedule;

// Daily health check at 2 AM
$scheduler->schedule(
    'log-health-check',
    CronSchedule::fromExpression('0 2 * * *'),
    function() {
        $command = new LogHealthCheckCommand();
        $input = new ConsoleInput(['logs:health-check'], new ConsoleOutput(), null);
        $exitCode = $command->execute($input);

        if ($exitCode !== ExitCode::SUCCESS) {
            // Alert operations team
            $this->alerting->sendAlert(
                level: AlertLevel::WARNING,
                message: 'Log health check failed'
            );
        }

        return ['exit_code' => $exitCode];
    }
);

Health Check Metrics

Monitor these metrics from health checks:

• Directory existence: All required log directories present
• Write permissions: Can write to log directories
• Disk space: >100MB free (warning), >10% free (healthy)
• Large files: Individual files >50MB (rotation issue)
• Total log size: Track growth trends

Automated Alerts

// Alert on health check failures
final readonly class LogHealthMonitor
{
    public function checkAndAlert(): void
    {
        $health = $this->runHealthCheck();

        if (!$health['directories_ok']) {
            $this->alerting->sendAlert(
                level: AlertLevel::CRITICAL,
                message: 'Log directories missing or inaccessible',
                context: $health['directory_issues']
            );
        }

        if ($health['disk_space_low']) {
            $this->alerting->sendAlert(
                level: AlertLevel::WARNING,
                message: "Low disk space: {$health['free_space']}",
                context: $health['disk_info']
            );
        }

        if ($health['large_files_count'] > 0) {
            $this->alerting->sendAlert(
                level: AlertLevel::INFO,
                message: "{$health['large_files_count']} log files exceed 50MB",
                context: $health['large_files']
            );
        }
    }
}

Docker/Container Deployment

Directory Structure

storage/
└── logs/
    ├── app/            # Application logs
    ├── debug/          # Debug logs (dev only)
    ├── security/       # Security events
    └── jobs/           # Background job logs

Docker Volume Configuration

# docker-compose.yml
services:
  php:
    volumes:
      # Host mount for logs (accessible from host for monitoring)
      - ./storage/logs:/var/www/html/storage/logs

Benefits of host mounts:

• Logs persist across container restarts
• Direct access from host for monitoring tools
• Backup integration without entering container
• Log rotation tools on host can access files

Permissions Setup

# Fix permissions for Docker container
chmod -R 777 storage/logs

# Or use Alpine container for one-time fix
docker run --rm -v $(pwd)/storage:/mnt alpine chmod -R 777 /mnt/logs

Production Container Configuration

# Dockerfile
FROM php:8.3-fpm

# Create log directories
RUN mkdir -p /var/www/html/storage/logs/app && \
    mkdir -p /var/www/html/storage/logs/security && \
    mkdir -p /var/www/html/storage/logs/jobs && \
    chown -R www-data:www-data /var/www/html/storage && \
    chmod -R 755 /var/www/html/storage/logs

Performance Considerations

Log Volume Management

High-volume scenarios:

// Use size-based rotation for predictable disk usage
$handler = RotatingFileHandler::withSizeRotation(
    logFile: '/var/www/html/storage/logs/high-volume.log',
    maxFileSize: Byte::fromMegabytes(100),
    maxFiles: 20,
    compress: true
);

// Separate handlers for different log types
$errorHandler = RotatingFileHandler::daily($errorLog)
    ->withMinLevel(LogLevel::ERROR);  // Errors only

$debugHandler = RotatingFileHandler::withSizeRotation($debugLog, Byte::fromMegabytes(50))
    ->withMinLevel(LogLevel::DEBUG);  // Debug info

Performance Benchmarks (from integration tests):

• 1000 log entries: <5 seconds
• Rotation overhead: <100ms per rotation
• Compression overhead: ~200ms for 100MB file

Memory Management

// For very high-volume logging, consider async logging
use App\Framework\Queue\Queue;

final readonly class AsyncLoggingHandler implements LogHandler
{
    public function handle(LogRecord $record): void
    {
        // Queue log writes for background processing
        $job = new WriteLogJob($record);
        $this->queue->push(JobPayload::immediate($job));
    }
}

Disk I/O Optimization

Best practices:

• Write logs synchronously for errors (data loss prevention)
• Buffer INFO/DEBUG logs for batch writes (performance)
• Use separate handlers for different log types (parallel writes)
• Monitor disk I/O metrics during rotation

Security Considerations

Log File Permissions

# Recommended permissions
chmod 755 storage/logs          # Directory: Read + execute for all, write for owner
chmod 644 storage/logs/*.log    # Files: Read for all, write for owner

Security guidelines:

• Never 777 permissions in production (security risk)
• Use 755 for directories (allows reading but prevents unauthorized writes)
• Use 644 for files (readable by monitoring tools, writable only by application)
• Separate user for web server and log access

Sensitive Data Logging

DO NOT log:

• Passwords (plaintext or hashed)
• API keys and secrets
• Credit card numbers
• Personal identification numbers (SSN, etc.)
• Session tokens

Safe logging patterns:

// ❌ Unsafe: Logs sensitive data
$logger->info('User login', [
    'password' => $password,  // NEVER log passwords
    'api_key' => $apiKey      // NEVER log keys
]);

// ✅ Safe: Logs only necessary information
$logger->info('User login', [
    'user_id' => $userId,
    'ip_address' => $ipAddress,
    'user_agent' => $userAgent
]);

// ✅ Safe: Mask sensitive data
$logger->info('Payment processed', [
    'card_last_four' => substr($cardNumber, -4),  // Only last 4 digits
    'amount' => $amount,
    'transaction_id' => $transactionId
]);

Security Event Logging

// Use framework's OWASP security logger for security events
use App\Framework\Security\OWASPSecurityLogger;
use App\Framework\Security\OWASPEventIdentifier;

$this->owaspLogger->logSecurityEvent(
    new SecurityEventType(OWASPEventIdentifier::AUTHN_LOGIN_FAILURE),
    request: $request,
    context: [
        'username' => $username,
        'ip_address' => $request->server->getRemoteAddr(),
        'failure_reason' => 'Invalid credentials'
    ]
);

Troubleshooting

Common Issues

Issue: Logs not rotating

Symptoms:

• Log file grows continuously
• No .1, .2 rotation files created

Diagnosis:

# Check file permissions
ls -la storage/logs/

# Check disk space
df -h

# Verify handler configuration
docker exec php php console.php logs:health-check --detailed

Solutions:

// Verify rotation configuration
$handler = RotatingFileHandler::daily($logFile, maxFiles: 30);

// Check if logs are actually being written
if (!file_exists($logFile)) {
    // Permission issue or incorrect path
}

// Manually trigger rotation for testing
touch($logFile, strtotime('yesterday'));
$handler->handle($logRecord);  // Should rotate now

Issue: Permission denied errors

Symptoms:

• Cannot write to log files
• Health check reports permission errors

Solutions:

# Fix permissions (development)
chmod -R 777 storage/logs

# Fix permissions (production - more secure)
chown -R www-data:www-data storage/logs
chmod -R 755 storage/logs
chmod -R 644 storage/logs/*.log

# Use health check auto-fix
docker exec php php console.php logs:health-check --fix-permissions

Issue: Disk space full

Symptoms:

• Health check reports low disk space
• Log rotation fails

Solutions:

# Check disk usage
du -sh storage/logs/*

# Find largest log files
find storage/logs -type f -exec ls -lh {} \; | sort -k5 -rh | head -10

# Manually clean old logs
find storage/logs -name "*.log.*" -mtime +30 -delete

# Increase compression for existing logs
find storage/logs -name "*.log.[0-9]*" ! -name "*.gz" -exec gzip {} \;

Prevention:

// Reduce retention period
$handler = RotatingFileHandler::daily($logFile, maxFiles: 14);  // 14 days instead of 30

// Reduce file size for size-based rotation
$handler = RotatingFileHandler::withSizeRotation(
    $logFile,
    maxFileSize: Byte::fromMegabytes(50),  // 50MB instead of 100MB
    maxFiles: 10
);

// Enable compression if not already
$handler = RotatingFileHandler::daily($logFile, compress: true);

Issue: Performance degradation

Symptoms:

• Slow request processing
• High disk I/O during logging

Solutions:

// Separate handlers by log type to parallelize writes
$errorHandler = RotatingFileHandler::daily('/var/www/html/storage/logs/errors.log')
    ->withMinLevel(LogLevel::ERROR);

$infoHandler = RotatingFileHandler::daily('/var/www/html/storage/logs/info.log')
    ->withMinLevel(LogLevel::INFO)
    ->withMaxLevel(LogLevel::WARNING);

// Consider async logging for high-volume scenarios
use App\Framework\Logging\Handlers\QueuedLogHandler;

$asyncHandler = new QueuedLogHandler($queue, $handler);

Integration with Framework Systems

Scheduler Integration

// Automated log rotation cleanup
$scheduler->schedule(
    'log-cleanup',
    CronSchedule::fromExpression('0 3 * * *'),  // Daily at 3 AM
    function() {
        // Remove logs older than retention policy
        $this->logCleanupService->cleanupExpiredLogs();

        return ['cleaned' => true];
    }
);

// Health monitoring
$scheduler->schedule(
    'log-health-check',
    IntervalSchedule::every(Duration::fromHours(6)),
    function() {
        $command = new LogHealthCheckCommand();
        $input = new ConsoleInput(['logs:health-check'], new ConsoleOutput(), null);

        return ['exit_code' => $command->execute($input)];
    }
);

Queue Integration

// Background log processing for high-volume scenarios
final class ProcessLogBatchJob
{
    public function handle(): void
    {
        $logBatch = $this->logBuffer->flush();

        foreach ($logBatch as $logRecord) {
            $this->handler->handle($logRecord);
        }
    }
}

Event System Integration

// Log important domain events
use App\Framework\Event\EventHandler;

#[EventHandler]
final readonly class UserRegistrationLogger
{
    public function handle(UserRegisteredEvent $event): void
    {
        $this->logger->info('User registered', [
            'user_id' => $event->userId,
            'email' => $event->email->getMasked(),
            'registration_method' => $event->method
        ]);
    }
}

Monitoring and Alerting

Metrics to Track

1. Log Volume Metrics:

   • Logs per hour/day
   • Growth rate trends
   • Rotation frequency

2. Performance Metrics:

   • Write latency (should be <10ms)
   • Rotation duration (should be <1s)
   • Disk I/O utilization

3. Health Metrics:

   • Disk space free percentage
   • Number of files exceeding size thresholds
   • Permission check failures

4. Error Metrics:

   • Error log volume (sudden spikes indicate issues)
   • Failed rotation attempts
   • Write permission errors

Alert Configuration

// Example monitoring integration
final readonly class LogMonitoringService
{
    public function collectMetrics(): array
    {
        return [
            'total_log_size' => $this->getTotalLogSize(),
            'log_file_count' => $this->getLogFileCount(),
            'largest_file_size' => $this->getLargestFileSize(),
            'disk_space_free' => $this->getDiskSpaceFree(),
            'rotation_count_24h' => $this->getRotationCount(Duration::fromHours(24))
        ];
    }

    public function checkThresholds(array $metrics): array
    {
        $alerts = [];

        if ($metrics['disk_space_free'] < 1_000_000_000) {  // <1GB
            $alerts[] = [
                'level' => 'critical',
                'message' => 'Log disk space critically low'
            ];
        }

        if ($metrics['largest_file_size'] > 500_000_000) {  // >500MB
            $alerts[] = [
                'level' => 'warning',
                'message' => 'Large log file detected (rotation issue?)'
            ];
        }

        return $alerts;
    }
}

Summary Checklist

Production Deployment

• Configure appropriate rotation strategy (daily/weekly/size-based)
• Set retention period based on compliance requirements
• Enable compression for rotated files
• Set production log level (INFO and above)
• Configure separate log files by severity (errors, security, etc.)
• Set up Docker volume mounts for log persistence
• Fix file permissions (755 directories, 644 files)
• Schedule health checks (daily recommended)
• Configure monitoring and alerting
• Test log rotation under load
• Document log access procedures for operations team

Development Setup

• Use DEBUG log level for development
• Disable compression for easier access
• Use shorter retention (7 days sufficient)
• Smaller file size limits for size-based rotation
• Regular cleanup to prevent disk space issues
• Test rotation locally before deploying

Ongoing Maintenance

• Monitor disk space weekly
• Review health check results daily
• Analyze log volume trends monthly
• Clean up old logs exceeding retention period
• Review and update alerting thresholds quarterly
• Test disaster recovery procedures (log restoration)
• Update documentation as logging patterns evolve

Additional Resources

• LogHealthCheckCommand: docs/logging/log-health-check-command.md
• RotatingFileHandler Implementation: src/Framework/Logging/Handlers/RotatingFileHandler.php
• Integration Tests: tests/Integration/Framework/Logging/LoggingInfrastructureTest.php
• OWASP Security Logging: docs/claude/security-patterns.md