michaelschiemer/docs/n-plus-one-detection.md

# N+1 Query Detection & Prevention

Comprehensive N+1 query detection system for the Custom PHP Framework.

## Overview

The N+1 Query Detection System automatically identifies and analyzes N+1 query problems - one of the most common performance killers in database-driven applications. N+1 problems occur when an application executes 1 query to load parent records, then N additional queries to load related data for each parent.

**Performance Impact**: Fixing N+1 problems can improve performance by **10-100x** by reducing hundreds of database queries to just 1-2.

## Architecture

```
┌─────────────────────┐
│  ProfilingConnection│ (Captures all queries)
└──────────┬──────────┘
           │
           v
┌──────────────────────┐
│    QueryLogger       │ (Collects query execution data)
└──────────┬───────────┘
           │
           v
┌──────────────────────┐
│  NPlusOneDetector    │ (Analyzes patterns, detects N+1)
└──────────┬───────────┘
           │
           v
┌───────────────────────┐
│ EagerLoadingAnalyzer  │ (Generates optimization strategies)
└──────────┬────────────┘
           │
           v
┌────────────────────────────┐
│ NPlusOneDetectionService   │ (Facade & Public API)
└────────────────────────────┘
```

## Core Components

### 1. QueryLog Value Object

Represents a single executed database query with full context.

```php
final readonly class QueryLog
{
    public function __construct(
        public string $sql,                   // SQL query
        public array $bindings,               // Bound parameters
        public float $executionTimeMs,        // Execution time in milliseconds
        public string $stackTrace,            // Full stack trace
        public int $rowCount = 0,            // Number of rows returned/affected
        public ?string $callerClass = null,   // Calling class
        public ?string $callerMethod = null,  // Calling method
        public int $callerLine = 0           // Line number
    ) {}

    // Pattern normalization for grouping similar queries
    public function getPattern(): string;

    // Query type detection
    public function isSelect(): bool;
    public function isInsert(): bool;
    public function isUpdate(): bool;
    public function isDelete(): bool;

    // Analysis helpers
    public function getTableName(): ?string;
    public function hasWhereClause(): bool;
    public function hasJoin(): bool;
    public function isSlow(float $thresholdMs = 100.0): bool;
}
```

**Pattern Normalization Example**:
```php
// Original SQL
"SELECT * FROM users WHERE id = 123 AND email = 'test@example.com'"

// Normalized Pattern
"SELECT * FROM users WHERE id = ? AND email = ?"
```

### 2. QueryPattern Value Object

Represents a repeated SQL pattern with N+1 detection logic.

```php
final readonly class QueryPattern
{
    public function __construct(
        public string $pattern,        // Normalized SQL pattern
        public array $queries          // array<QueryLog>
    ) {}

    // N+1 Detection
    public function isPotentialNPlusOne(): bool;
    public function getNPlusOneSeverity(): int;  // 0-10 scale
    public function getSeverityLevel(): string;  // CRITICAL, HIGH, MEDIUM, LOW, INFO

    // Statistics
    public function getExecutionCount(): int;
    public function getTotalExecutionTimeMs(): float;
    public function getAverageExecutionTimeMs(): float;

    // Analysis
    public function getTableName(): ?string;
    public function hasConsistentCaller(): bool;
    public function getCallerLocations(): array;
}
```

**N+1 Detection Criteria**:
- Executed >5 times (configurable)
- Has WHERE clause (filtering by ID)
- No JOIN present (could have been optimized)
- Is SELECT query

**Severity Scoring** (0-10):
- **Execution count**: Up to 4 points
- **Performance impact**: Up to 3 points
- **Consistent caller**: Up to 2 points
- **Total time impact**: Up to 1 point

**Severity Levels**:
- **CRITICAL**: Score >= 8 (immediate action required)
- **HIGH**: Score >= 6 (fix soon)
- **MEDIUM**: Score >= 4 (optimization opportunity)
- **LOW**: Score >= 2 (minor issue)
- **INFO**: Score < 2 (informational)

### 3. NPlusOneDetection Value Object

Detection result with recommendations and performance impact estimates.

```php
final readonly class NPlusOneDetection
{
    public function __construct(
        public QueryPattern $pattern,       // Detected pattern
        public string $recommendation,      // How to fix
        public array $metadata = []        // Additional context
    ) {}

    public function getSeverityLevel(): string;
    public function getSeverityScore(): int;
    public function isCritical(): bool;
    public function getPerformanceImpact(): string;
}
```

**Performance Impact Example**:
```
"Reducing 50 queries to ~2 could save ~96% (500ms → ~20ms)"
```

### 4. EagerLoadingStrategy

Optimization strategy with code examples.

```php
final readonly class EagerLoadingStrategy
{
    public function __construct(
        public string $tableName,            // Affected table
        public string $relationshipType,     // belongsTo, hasMany, belongsToMany
        public string $codeExample,          // Before/after code
        public int $priority,                // 0-200 priority score
        public string $estimatedImprovement, // Performance estimate
        public array $affectedLocations     // Where to apply fix
    ) {}
}
```

**Relationship Type Detection**:
- `WHERE.*_id = ?` → **belongsTo** (many-to-one)
- `JOIN.*_.*ON` → **belongsToMany** (many-to-many)
- Default → **hasMany** (one-to-many)

**Code Example Template**:
```php
// Before (N+1 Problem):
$items = $repository->findAll();
foreach ($items as $item) {
    $related = $item->posts; // Triggers separate query
}

// After (Eager Loading):
$items = $repository->findWithRelations(['*'], ['posts']);
// All posts loaded in 1-2 queries total
```

## NPlusOneDetectionService API

Main facade for all N+1 detection operations.

### Start/Stop Logging

```php
$service->startLogging();  // Enable query logging
$service->stopLogging();   // Disable query logging
```

### Analysis Methods

```php
// Analyze all logged queries
$result = $service->analyze();
/*
Returns:
[
    'detections' => array<NPlusOneDetection>,
    'strategies' => array<EagerLoadingStrategy>,
    'statistics' => [
        'total_queries' => 150,
        'total_patterns' => 12,
        'n_plus_one_patterns' => 3,
        'n_plus_one_queries' => 89,
        'n_plus_one_percentage' => 59.3,
        'time_wasted_percentage' => 78.5
    ]
]
*/

// Generate formatted text report
$report = $service->analyzeAndReport();

// Quick check for N+1 problems
$hasProblems = $service->hasNPlusOneProblems(); // boolean

// Get only critical problems (severity >= 8)
$critical = $service->getCriticalProblems(); // array<NPlusOneDetection>
```

### Profiling Specific Code

```php
// Profile a specific code block
$analysis = $service->profile(function() {
    // Your code here
    $users = $userRepository->findAll();

    foreach ($users as $user) {
        $posts = $user->getPosts(); // Potential N+1
    }

    return $users;
});

/*
Returns analysis + execution info:
[
    'detections' => [...],
    'strategies' => [...],
    'statistics' => [...],
    'execution_time_ms' => 245.67,
    'callback_result' => $users
]
*/
```

### Query Statistics

```php
// Get general query statistics
$stats = $service->getQueryStatistics();
/*
[
    'total_queries' => 150,
    'total_time_ms' => 345.6,
    'average_time_ms' => 2.304,
    'select_count' => 120,
    'insert_count' => 15,
    'update_count' => 10,
    'delete_count' => 5,
    'slow_queries' => 3,
    'unique_patterns' => 25
]
*/

// Get all logged queries
$logs = $service->getQueryLogs(); // array<QueryLog>

// Clear query logs
$service->clearLogs();
```

## Console Command

### Usage

```bash
# Profile specific code
php console.php detect:n-plus-one --profile=UserController::index

# Generate full report
php console.php detect:n-plus-one --report

# Show only critical problems
php console.php detect:n-plus-one --critical-only

# Show usage help
php console.php detect:n-plus-one
```

### Interactive Profiling Workflow

```bash
$ php console.php detect:n-plus-one --profile=UserService::getUsers

Profiling: UserService::getUsers
============================================================

Query logging started. Execute your code now...
Note: This command only starts logging. You need to execute the code separately.
Press Enter when done...

[Execute your code in browser/tests]
[Press Enter]

=== N+1 Query Detection Report ===

Query Statistics:
  Total queries: 52
  Total patterns: 8
  Total time: 234.50ms

N+1 Problems Detected:
  N+1 patterns: 2
  N+1 queries: 45 (86.5% of total)
  Time wasted: 198.30ms (84.5% of total)

Detected Issues:

[1] CRITICAL - posts
    Score: 9/10
    Executions: 40
    Total time: 180.00ms
    Impact: Reducing 40 queries to ~2 could save ~95% (180ms → ~9ms)

    Recommendation:
    N+1 Query detected in table 'posts'

    Location: UserService::getUsers:42

    Recommended fix:
    1. Use eager loading instead of lazy loading
       Example: $repository->findWithRelations($ids, ['posts'])

    2. Or use batch loading:
       Example: $repository->batchLoad($ids)

    3. Or add JOIN to initial query:
       Example: SELECT * FROM parent LEFT JOIN posts ON ...

[2] HIGH - comments
    Score: 7/10
    Executions: 5
    Total time: 18.30ms
    Impact: Reducing 5 queries to ~2 could save ~60% (18.30ms → ~7.32ms)

=== Eager Loading Opportunities ===

Found 2 opportunities:

[1] posts (Priority: 150)
    Relationship Type: hasMany
    Estimated Improvement: Reducing 40 queries to ~2 could save ~95% (180ms → ~9ms)
    Affected Locations:
      - UserService::getUsers:42

    Code Example:
    // Before (N+1 Problem):
    $items = $repository->findAll();
    foreach ($items as $item) {
        $related = $item->posts; // Triggers separate query
    }

    // After (Eager Loading):
    $items = $repository->findWithRelations(['*'], ['posts']);
    // This will load all related posts in one query
```

## Automatic Integration with ProfilingConnection

The N+1 Detection System automatically integrates with the framework's `ProfilingConnection` when profiling is enabled.

### Setup in DI Container

```php
// NPlusOneDetectionServiceInitializer handles setup automatically
final readonly class NPlusOneDetectionServiceInitializer
{
    #[Initializer]
    public function __invoke(Container $container): NPlusOneDetectionService
    {
        $queryLogger = new QueryLogger();
        $detector = new NPlusOneDetector();
        $analyzer = new EagerLoadingAnalyzer();

        $service = new NPlusOneDetectionService(
            $queryLogger,
            $detector,
            $analyzer,
            $this->logger
        );

        // Automatic integration
        $this->setupProfilingIntegration($queryLogger);

        return $service;
    }
}
```

### How It Works

1. **ProfilingConnection** captures every query execution
2. **NPlusOneQueryLoggerIntegration** forwards query data to **QueryLogger**
3. **QueryLogger** collects queries (when enabled)
4. **NPlusOneDetectionService** analyzes patterns on demand

## Common N+1 Patterns

### Pattern 1: User Posts (hasMany)

**Problem**:
```php
$users = $userRepository->findAll(); // 1 query

foreach ($users as $user) {
    $posts = $user->getPosts(); // N queries (one per user)
}
```

**Solution**:
```php
$users = $userRepository->findWithRelations(['*'], ['posts']); // 2 queries total

foreach ($users as $user) {
    $posts = $user->getPosts(); // Already loaded
}
```

### Pattern 2: Post Author (belongsTo)

**Problem**:
```php
$posts = $postRepository->findAll(); // 1 query

foreach ($posts as $post) {
    $author = $post->getAuthor(); // N queries
}
```

**Solution**:
```php
$posts = $queryBuilder
    ->select('posts.*', 'users.name as author_name')
    ->leftJoin('users', 'posts.user_id', '=', 'users.id')
    ->get(); // 1 query with JOIN
```

### Pattern 3: Many-to-Many Tags

**Problem**:
```php
$posts = $postRepository->findAll(); // 1 query

foreach ($posts as $post) {
    $tags = $post->getTags(); // N queries via pivot table
}
```

**Solution**:
```php
$posts = $postRepository->findWithRelations(['*'], ['tags']); // 2 queries
// Query 1: SELECT * FROM posts
// Query 2: SELECT * FROM tags JOIN post_tag ...
```

## Configuration

### NPlusOneDetector Configuration

```php
$detector = new NPlusOneDetector(
    minExecutionCount: 5,      // Minimum executions to consider N+1
    minSeverityScore: 4.0      // Minimum severity to include in results
);
```

**Tuning Guidelines**:
- **Low traffic dev**: `minExecutionCount: 3` (catch smaller issues)
- **High traffic prod**: `minExecutionCount: 10` (reduce noise)
- **All issues**: `minSeverityScore: 0.0`
- **Critical only**: `minSeverityScore: 8.0`

## Performance Characteristics

### Detection Overhead

- **Logging overhead**: <1ms per query
- **Pattern analysis**: <10ms for 100 queries
- **Report generation**: <50ms for complex analysis

### Memory Usage

- ~1KB per logged query
- ~100 queries = ~100KB memory

### Typical Performance Gains

| N+1 Severity | Queries Before | Queries After | Performance Improvement |
|--------------|----------------|---------------|-------------------------|
| CRITICAL (50+ queries) | 52 | 2 | 10-50x faster |
| HIGH (20-50 queries) | 25 | 2-3 | 5-10x faster |
| MEDIUM (10-20 queries) | 15 | 2-3 | 3-5x faster |

## Best Practices

### 1. Use Profiling in Development

```php
// Enable N+1 detection in development
if ($environment === 'development') {
    $service->startLogging();

    // Your code

    if ($service->hasNPlusOneProblems()) {
        $report = $service->analyzeAndReport();
        error_log($report);
    }
}
```

### 2. Profile Critical User Flows

```php
// Profile specific user actions
$analysis = $service->profile(function() {
    // Critical user flow
    return $dashboardController->index($request);
});

foreach ($analysis['detections'] as $detection) {
    if ($detection->isCritical()) {
        // Alert development team
        $this->alerting->send("Critical N+1 in dashboard: " . $detection->pattern->getTableName());
    }
}
```

### 3. Integrate with CI/CD

```bash
# In test suite
php console.php detect:n-plus-one --critical-only

# Fail build if critical N+1 problems detected
if [ $? -ne 0 ]; then
    echo "Critical N+1 problems detected!"
    exit 1
fi
```

### 4. Monitor Production

```php
// Periodic N+1 detection in production (low overhead)
$scheduler->schedule(
    'n-plus-one-check',
    CronSchedule::fromExpression('0 */6 * * *'), // Every 6 hours
    function() use ($service) {
        $service->startLogging();

        // Run sample of typical operations
        $this->runSampleOperations();

        $service->stopLogging();

        $critical = $service->getCriticalProblems();

        if (!empty($critical)) {
            $this->sendAlertToOps($critical);
        }
    }
);
```

### 5. Document Known N+1 Patterns

```php
// Mark intentional N+1 patterns with comments
public function getUserPosts(UserId $userId): array
{
    // INTENTIONAL N+1: Posts loaded on-demand for admin dashboard
    // Trade-off: Reduces memory usage for large datasets
    // Acceptable because: Admin views typically show <10 users

    return $this->postRepository->findByUserId($userId);
}
```

## Troubleshooting

### Issue: No N+1 Problems Detected

**Causes**:
- Logging not enabled (`$service->startLogging()`)
- Too few query executions (< minExecutionCount)
- All queries use JOINs (already optimized)
- Severity threshold too high

**Solution**:
```php
// Check if logging is enabled
if (!$this->queryLogger->isEnabled()) {
    $service->startLogging();
}

// Lower thresholds temporarily
$detector = new NPlusOneDetector(
    minExecutionCount: 3,
    minSeverityScore: 0.0
);
```

### Issue: False Positives

**Causes**:
- Batch loading patterns detected as N+1
- Intentional on-demand loading for performance
- Pagination queries

**Solution**:
```php
// Review severity score - focus on CRITICAL/HIGH only
$critical = $service->getCriticalProblems();

// Or increase thresholds
$detector = new NPlusOneDetector(
    minExecutionCount: 10,  // More conservative
    minSeverityScore: 6.0   // HIGH and above only
);
```

### Issue: Performance Impact of Detection

**Causes**:
- Logging enabled in production
- Very high query volume

**Solution**:
```php
// Only enable in development/staging
if ($environment !== 'production') {
    $service->startLogging();
}

// Or use sampling in production
if (random_int(1, 100) === 1) { // 1% sample
    $service->startLogging();
}
```

## Framework Integration Patterns

### With EntityManager

```php
class UserController
{
    public function __construct(
        private readonly UserRepository $repository,
        private readonly NPlusOneDetectionService $nPlusOneService
    ) {}

    public function index(): ViewResult
    {
        // Enable detection for this request
        $this->nPlusOneService->startLogging();

        $users = $this->repository->findAll();

        // EntityManager queries are automatically logged

        if ($this->nPlusOneService->hasNPlusOneProblems()) {
            $this->logger->warning('N+1 detected in UserController::index');
        }

        return new ViewResult('users/index', ['users' => $users]);
    }
}
```

### With Test Suite

```php
describe('UserService', function () {
    beforeEach(function () {
        $this->nPlusOneService = $this->container->get(NPlusOneDetectionService::class);
        $this->nPlusOneService->startLogging();
    });

    afterEach(function () {
        // Assert no N+1 problems in tests
        $critical = $this->nPlusOneService->getCriticalProblems();
        expect($critical)->toBeEmpty();
    });

    it('loads users without N+1', function () {
        $users = $this->userService->getAllUsersWithPosts();

        // Assertion happens in afterEach
    });
});
```

### With Events

```php
// Listen for N+1 detection
$eventDispatcher->listen(NPlusOneDetectedEvent::class, function($event) {
    $detection = $event->detection;

    if ($detection->isCritical()) {
        $this->metrics->increment('n_plus_one.critical', [
            'table' => $detection->pattern->getTableName()
        ]);
    }
});
```

## Summary

The N+1 Query Detection System provides:

✅ **Automatic Detection** - Identifies N+1 patterns automatically
✅ **Severity Scoring** - Prioritizes issues by impact (0-10 scale)
✅ **Actionable Recommendations** - Provides fix strategies with code examples
✅ **Integration** - Works seamlessly with ProfilingConnection
✅ **Console Interface** - CLI tools for profiling and reporting
✅ **Minimal Overhead** - <1ms per query logging overhead
✅ **Production-Ready** - Suitable for sampling in production environments

**Typical Results**: 10-100x performance improvement by reducing hundreds of queries to 1-2.