michaelschiemer/docs/performance/index-optimization.md

# Database Index Optimization

Comprehensive guide for database index analysis and optimization in the Custom PHP Framework.

## Overview

The Index Optimization system provides automated tools for:
- **Index Usage Analysis**: Track real index usage statistics
- **Unused Index Detection**: Find indexes that waste storage and slow writes
- **Smart Recommendations**: Generate composite index suggestions based on query patterns
- **Automatic Migration Generation**: Create migration files for index optimizations
- **Performance Metrics**: Measure index effectiveness and query speedup

## Core Components

### 1. IndexAnalyzer
Core service for analyzing database index usage and effectiveness.

**Capabilities**:
- Parse EXPLAIN output (MySQL, PostgreSQL, SQLite)
- Detect actual index usage in queries
- Get all indexes for a table with metadata
- Multi-database support with driver-specific optimizations

**Usage**:
```php
use App\Framework\Database\Indexing\IndexAnalyzer;

$analyzer = $container->get(IndexAnalyzer::class);

// Get all indexes for a table
$indexes = $analyzer->getTableIndexes('users');

foreach ($indexes as $index) {
    echo "Index: {$index['name']}\n";
    echo "Columns: " . implode(', ', $index['columns']) . "\n";
    echo "Type: {$index['type']->value}\n";
    echo "Unique: " . ($index['is_unique'] ? 'Yes' : 'No') . "\n";
}

// Analyze query for index usage
$sql = 'SELECT * FROM users WHERE email = ? AND status = ?';
$analysis = $analyzer->analyzeQuery($sql);

echo "Indexes used: " . count($analysis['indexes_used']) . "\n";
echo "Key type: {$analysis['key_type']}\n";
echo "Rows examined: {$analysis['rows_examined']}\n";
echo "Using filesort: " . ($analysis['using_filesort'] ? 'Yes' : 'No') . "\n";
```

### 2. IndexUsageTracker
Tracks real index usage statistics over time using cache.

**Capabilities**:
- Record index usage for queries
- Calculate index selectivity and efficiency
- Track usage count and last used timestamp
- Generate usage metrics with Value Objects

**Usage**:
```php
use App\Framework\Database\Indexing\IndexUsageTracker;
use App\Framework\Database\Indexing\ValueObjects\IndexName;

$tracker = $container->get(IndexUsageTracker::class);

// Record usage for a query
$tracker->recordUsage('SELECT * FROM users WHERE email = ?', 'users');

// Get usage metrics for specific index
$indexName = new IndexName('idx_users_email');
$metrics = $tracker->getUsageMetrics($indexName, 'users');

if ($metrics) {
    echo "Usage count: {$metrics->usageCount}\n";
    echo "Efficiency: " . number_format($metrics->getEfficiency() * 100, 2) . "%\n";
    echo "Selectivity: " . number_format($metrics->selectivity, 2) . "\n";
    echo "Days since last use: {$metrics->getDaysSinceLastUse()}\n";
}

// Get all usage metrics for a table
$allMetrics = $tracker->getTableUsageMetrics('users');
```

### 3. UnusedIndexDetector
Detects unused, duplicate, and redundant indexes.

**Capabilities**:
- Find unused indexes (configurable days threshold)
- Detect duplicate indexes (identical column coverage)
- Find redundant indexes (prefix patterns)
- Generate DROP statements for cleanup
- Estimate space savings

**Usage**:
```php
use App\Framework\Database\Indexing\UnusedIndexDetector;

$detector = $container->get(UnusedIndexDetector::class);

// Find unused indexes (not used in last 30 days)
$unusedIndexes = $detector->findUnusedIndexes('users', daysThreshold: 30);

foreach ($unusedIndexes as $index) {
    echo "Unused: {$index['index_name']}\n";
    echo "Columns: " . implode(', ', $index['columns']) . "\n";
    echo "Last used: {$index['last_used_days_ago']} days ago\n";
    echo "Reason: {$index['reason']}\n";
}

// Find duplicate indexes
$duplicates = $detector->findDuplicateIndexes('users');

// Find redundant indexes (prefix pattern)
$redundant = $detector->findRedundantIndexes('users');

// Get comprehensive report
$report = $detector->getUnusedIndexReport('users', daysThreshold: 30);

echo "Total removable: {$report['total_removable']}\n";
echo "Estimated space savings: {$report['estimated_space_savings']}\n";

// Generate DROP statements
$dropStatements = $detector->generateDropStatements('users');

foreach ($dropStatements as $sql) {
    echo "{$sql}\n";
}
```

### 4. CompositeIndexGenerator
Generates smart composite index recommendations based on query patterns.

**Capabilities**:
- Analyze slow queries for index opportunities
- Suggest composite indexes (WHERE + ORDER BY columns)
- Detect full table scans needing indexes
- Estimate query speedup
- Prioritize recommendations (CRITICAL/HIGH/MEDIUM/LOW)

**Usage**:
```php
use App\Framework\Database\Indexing\CompositeIndexGenerator;

$generator = $container->get(CompositeIndexGenerator::class);

// Generate recommendations for a table
$recommendations = $generator->generateRecommendations('users');

foreach ($recommendations as $recommendation) {
    echo "Priority: {$recommendation->priority->value}\n";
    echo "Index: {$recommendation->getIndexName()->toString()}\n";
    echo "Columns: {$recommendation->getColumnsString()}\n";
    echo "Reason: {$recommendation->reason}\n";
    echo "Estimated speedup: {$recommendation->estimatedSpeedup}x\n";
    echo "Affected queries: {$recommendation->affectedQueries}\n";
    echo "\n";
}
```

### 5. IndexMigrationGenerator
Generates database migration files for index optimizations.

**Capabilities**:
- Generate ADD INDEX migrations
- Generate DROP INDEX migrations
- Generate comprehensive optimization migrations (add + remove)
- Auto-save migrations with timestamp
- Include UP and DOWN methods for rollback

**Usage**:
```php
use App\Framework\Database\Indexing\IndexMigrationGenerator;

$migrationGen = $container->get(IndexMigrationGenerator::class);

// Generate migration for adding recommended indexes
$recommendations = [/* IndexRecommendation objects */];
$migration = $migrationGen->generateAddIndexMigration($recommendations, 'users');

echo $migration; // PHP migration file content

// Generate migration for removing unused indexes
$unusedIndexes = [
    ['index_name' => 'idx_users_old', 'columns' => ['old_column']]
];
$migration = $migrationGen->generateRemoveIndexMigration($unusedIndexes, 'users');

// Generate comprehensive optimization migration
$migration = $migrationGen->generateOptimizationMigration(
    toAdd: $recommendations,
    toRemove: $unusedIndexes,
    tableName: 'users'
);

// Save migration to file
$path = $migrationGen->saveMigration($migration);
echo "Migration saved to: {$path}\n";
```

### 6. IndexOptimizationService
Facade service combining all index optimization components.

**Capabilities**:
- Complete table analysis (unused + recommendations)
- Generate optimization migrations automatically
- Index statistics dashboard
- High-priority recommendations across multiple tables
- Health check for optimization opportunities

**Usage**:
```php
use App\Framework\Database\Indexing\IndexOptimizationService;

$service = $container->get(IndexOptimizationService::class);

// Complete table analysis
$analysis = $service->analyzeTable('users', unusedDaysThreshold: 30);

echo "Current indexes: " . count($analysis['current_indexes']) . "\n";
echo "Unused indexes: {$analysis['total_removable']}\n";
echo "Recommended indexes: {$analysis['total_recommended']}\n";
echo "Space savings: {$analysis['estimated_space_savings']}\n";

// Generate and save optimization migration
$migrationPath = $service->generateOptimizationMigration('users');
echo "Migration created: {$migrationPath}\n";

// Get index statistics
$stats = $service->getIndexStatistics('users');

// Get high-priority recommendations for multiple tables
$tables = ['users', 'orders', 'products'];
$highPriority = $service->getHighPriorityRecommendations($tables);

// Health check
$healthCheck = $service->healthCheck($tables, unusedDaysThreshold: 30);

if ($healthCheck['requires_attention']) {
    echo "⚠️  Optimization required:\n";
    echo "  - Tables with unused indexes: " .
         count($healthCheck['tables_with_unused_indexes']) . "\n";
    echo "  - Total removable: {$healthCheck['total_removable_indexes']}\n";
    echo "  - Total recommended: {$healthCheck['total_recommended_indexes']}\n";
}
```

## Console Commands

### Analyze Indexes
```bash
# Analyze specific table
php console.php db:analyze-indexes users

# Output:
# 🔍 Analyzing indexes for table: users
#
# 📊 Current Indexes (5 total):
#   - PRIMARY (PRIMARY): id
#   - idx_users_email (BTREE): email
#   - idx_users_status (BTREE): status
#   - idx_users_created_at (BTREE): created_at
#   - idx_users_email_status (BTREE): email, status
#
# 🗑️  Unused Indexes (2 total):
#   - idx_users_old_column: old_column (unused for 120 days)
#   - idx_users_deprecated: deprecated_field (unused for 90 days)
#
# 💡 Recommended Indexes (1 total):
#   - [HIGH] idx_users_status_created_at: status, created_at
#     Reason: WHERE status + ORDER BY created_at
#     Estimated speedup: 5.0x
#
# 📈 Summary:
#   - Removable indexes: 2
#   - Recommended indexes: 1
#   - Estimated space savings: 10 MB
#
# 💾 To generate migration, run:
#    php console.php db:generate-index-migration users
```

## Value Objects

### IndexName
Validated index name (1-64 characters, alphanumeric + underscore).

```php
use App\Framework\Database\Indexing\ValueObjects\IndexName;

$indexName = new IndexName('idx_users_email');
echo $indexName->toString(); // "idx_users_email"
```

### IndexType
Enum representing database index types.

```php
use App\Framework\Database\Indexing\ValueObjects\IndexType;

$type = IndexType::BTREE;
echo $type->getDescription(); // "Balanced tree index - good for range queries"

// Database-specific support check
$isSupported = $type->isSupported('mysql'); // true
```

**Supported Types**:
- `BTREE`: Balanced tree (default, all databases)
- `HASH`: Hash index (MySQL, PostgreSQL)
- `FULLTEXT`: Full-text search (MySQL)
- `SPATIAL`: Geographic data (MySQL)
- `GIN`: Generalized Inverted Index (PostgreSQL)
- `GIST`: Generalized Search Tree (PostgreSQL)
- `BRIN`: Block Range Index (PostgreSQL)
- `PRIMARY`: Primary key
- `UNIQUE`: Unique constraint

### IndexUsageMetrics
Statistics about index usage and effectiveness.

```php
use App\Framework\Database\Indexing\ValueObjects\IndexUsageMetrics;

$metrics = new IndexUsageMetrics(
    indexName: new IndexName('idx_users_email'),
    tableName: 'users',
    usageCount: 15234,
    scanCount: 15234,
    selectivity: 0.95,
    rowsExamined: 152340,
    rowsReturned: 15234,
    lastUsed: new DateTimeImmutable('2025-01-19 14:30:00'),
    createdAt: new DateTimeImmutable('2024-12-01 10:00:00')
);

// Computed metrics
$efficiency = $metrics->getEfficiency(); // 0.10 (10% of examined rows returned)
$avgScanSize = $metrics->getAverageScanSize(); // 10 rows per scan
$daysSinceLastUse = $metrics->getDaysSinceLastUse(); // 0
$isUnused = $metrics->isUnused(daysThreshold: 30); // false
```

### IndexRecommendation
Recommendation for creating or optimizing an index.

```php
use App\Framework\Database\Indexing\ValueObjects\IndexRecommendation;
use App\Framework\Database\Indexing\ValueObjects\IndexType;
use App\Framework\Database\Indexing\ValueObjects\RecommendationPriority;

$recommendation = new IndexRecommendation(
    tableName: 'users',
    columns: ['status', 'created_at'],
    indexType: IndexType::BTREE,
    reason: 'WHERE status + ORDER BY created_at; Query uses filesort',
    priority: RecommendationPriority::HIGH,
    estimatedSpeedup: 5.0,
    affectedQueries: 100
);

$indexName = $recommendation->getIndexName(); // IndexName("idx_users_status_created_at")
$isComposite = $recommendation->isComposite(); // true
$array = $recommendation->toArray(); // Full array representation
```

### RecommendationPriority
Priority levels for index recommendations.

```php
use App\Framework\Database\Indexing\ValueObjects\RecommendationPriority;

// Auto-detect priority from metrics
$priority = RecommendationPriority::fromMetrics(
    speedup: 15.0,
    affectedQueries: 250
); // CRITICAL (>10x speedup or >100 affected queries)

// Priority levels:
// - CRITICAL: >10x speedup or >100 affected queries
// - HIGH: >5x speedup or >50 affected queries
// - MEDIUM: >2x speedup or >20 affected queries
// - LOW: <2x speedup or <20 affected queries

echo $priority->value; // "critical"
echo $priority->getColor(); // "red"
```

## Best Practices

### 1. Regular Index Analysis
Run index analysis monthly or after major feature deployments:

```bash
# Analyze all critical tables
php console.php db:analyze-indexes users
php console.php db:analyze-indexes orders
php console.php db:analyze-indexes products
```

### 2. Unused Days Threshold
- **Development**: 7 days
- **Staging**: 14 days
- **Production**: 30-90 days (conservative)

### 3. Index Naming Convention
Generated index names follow pattern: `idx_{table}_{column1}_{column2}`

### 4. Composite Index Column Order
**Rule**: WHERE columns first, ORDER BY columns second

```sql
-- Query pattern:
SELECT * FROM users WHERE status = 'active' ORDER BY created_at DESC

-- Optimal index:
CREATE INDEX idx_users_status_created_at ON users(status, created_at)
```

### 5. Migration Safety
Always review generated migrations before running:

```bash
# Generate migration
php console.php db:generate-index-migration users

# Review file
cat migrations/20250119_optimize_indexes_for_users.php

# Apply migration
php console.php db:migrate
```

### 6. Monitor Index Effectiveness
Track index usage metrics regularly:

```php
$metrics = $tracker->getTableUsageMetrics('users');

foreach ($metrics as $metric) {
    if ($metric->getEfficiency() < 0.1) {
        // Index returns <10% of examined rows - might need optimization
        $this->logger->warning("Low efficiency index", [
            'index' => $metric->indexName->toString(),
            'efficiency' => $metric->getEfficiency()
        ]);
    }
}
```

### 7. Avoid Over-Indexing
**Costs of indexes**:
- Storage space (5-10% of table size per index)
- INSERT/UPDATE/DELETE slowdown
- Maintenance overhead

**Guidelines**:
- Limit to 5-7 indexes per table
- Focus on high-traffic queries
- Remove unused indexes regularly

## Integration with Existing Framework

### ProfilingDashboard Integration
```php
use App\Framework\Database\Profiling\ProfilingDashboard;
use App\Framework\Database\Indexing\IndexOptimizationService;

$dashboard = $container->get(ProfilingDashboard::class);
$indexService = $container->get(IndexOptimizationService::class);

// Generate comprehensive performance report
$report = $dashboard->generateReport();

// For each slow query, check if index would help
foreach ($report->slowQueries as $slowQuery) {
    $tableName = $this->extractTableName($slowQuery->query);
    $recommendations = $indexService->analyzeTable($tableName);

    if ($recommendations['total_recommended'] > 0) {
        echo "Index optimization available for {$tableName}\n";
    }
}
```

### SlowQueryDetector Integration
```php
use App\Framework\Database\Profiling\SlowQueryDetector;

$detector = $container->get(SlowQueryDetector::class);

// For N+1 query patterns, suggest composite indexes
$patterns = $detector->detectSlowQueryPatterns();

foreach ($patterns as $pattern) {
    if ($pattern->type === 'N_PLUS_ONE') {
        // Analyze and suggest composite index
        $recommendations = $generator->generateRecommendations($pattern->tableName);
    }
}
```

## Performance Characteristics

**Index Analysis**:
- **Typical Time**: <100ms per table
- **Memory Usage**: <10MB for most tables
- **Scalability**: Linear with table size

**Usage Tracking**:
- **Overhead**: <1ms per query
- **Cache Storage**: ~1KB per index
- **TTL**: 30 days (configurable)

**Migration Generation**:
- **Time**: <50ms for typical table
- **Output Size**: 1-2KB per migration

## Troubleshooting

### Issue: "No recommendations found"
**Cause**: No slow queries or all queries already optimized
**Solution**: Run queries for longer period to collect data

### Issue: "Index marked as unused but is actually used"
**Cause**: Usage tracking not enabled or cache cleared
**Solution**: Enable usage tracking for all queries:

```php
// In query execution interceptor
$this->indexUsageTracker->recordUsage($sql, $tableName);
```

### Issue: "Migration generation fails"
**Cause**: Invalid index name or table name
**Solution**: Check IndexName validation (alphanumeric + underscore, max 64 chars)

### Issue: "Duplicate index recommendations"
**Cause**: Similar query patterns analyzed multiple times
**Solution**: Deduplication is automatic - review merged recommendations

## Summary

The Index Optimization system provides:
- ✅ **Automated index analysis** with EXPLAIN parsing
- ✅ **Usage tracking** with cache-based metrics
- ✅ **Unused index detection** (unused/duplicate/redundant)
- ✅ **Smart recommendations** for composite indexes
- ✅ **Automatic migration generation** with rollback support
- ✅ **Console commands** for DBA workflows
- ✅ **Framework integration** with ProfilingDashboard and SlowQueryDetector
- ✅ **Multi-database support** (MySQL, PostgreSQL, SQLite)

**Framework Compliance**:
- Value Objects for type safety
- Readonly classes for immutability
- PSR-12 code style
- Comprehensive Pest tests
- Production-ready error handling