michaelschiemer/docs/performance/index-optimization.md

Database Index Optimization

Comprehensive guide for database index analysis and optimization in the Custom PHP Framework.

Overview

The Index Optimization system provides automated tools for:

• Index Usage Analysis: Track real index usage statistics
• Unused Index Detection: Find indexes that waste storage and slow writes
• Smart Recommendations: Generate composite index suggestions based on query patterns
• Automatic Migration Generation: Create migration files for index optimizations
• Performance Metrics: Measure index effectiveness and query speedup

Core Components

1. IndexAnalyzer

Core service for analyzing database index usage and effectiveness.

Capabilities:

• Parse EXPLAIN output (MySQL, PostgreSQL, SQLite)
• Detect actual index usage in queries
• Get all indexes for a table with metadata
• Multi-database support with driver-specific optimizations

Usage:

use App\Framework\Database\Indexing\IndexAnalyzer;

$analyzer = $container->get(IndexAnalyzer::class);

// Get all indexes for a table
$indexes = $analyzer->getTableIndexes('users');

foreach ($indexes as $index) {
    echo "Index: {$index['name']}\n";
    echo "Columns: " . implode(', ', $index['columns']) . "\n";
    echo "Type: {$index['type']->value}\n";
    echo "Unique: " . ($index['is_unique'] ? 'Yes' : 'No') . "\n";
}

// Analyze query for index usage
$sql = 'SELECT * FROM users WHERE email = ? AND status = ?';
$analysis = $analyzer->analyzeQuery($sql);

echo "Indexes used: " . count($analysis['indexes_used']) . "\n";
echo "Key type: {$analysis['key_type']}\n";
echo "Rows examined: {$analysis['rows_examined']}\n";
echo "Using filesort: " . ($analysis['using_filesort'] ? 'Yes' : 'No') . "\n";

2. IndexUsageTracker

Tracks real index usage statistics over time using cache.

Capabilities:

• Record index usage for queries
• Calculate index selectivity and efficiency
• Track usage count and last used timestamp
• Generate usage metrics with Value Objects

Usage:

use App\Framework\Database\Indexing\IndexUsageTracker;
use App\Framework\Database\Indexing\ValueObjects\IndexName;

$tracker = $container->get(IndexUsageTracker::class);

// Record usage for a query
$tracker->recordUsage('SELECT * FROM users WHERE email = ?', 'users');

// Get usage metrics for specific index
$indexName = new IndexName('idx_users_email');
$metrics = $tracker->getUsageMetrics($indexName, 'users');

if ($metrics) {
    echo "Usage count: {$metrics->usageCount}\n";
    echo "Efficiency: " . number_format($metrics->getEfficiency() * 100, 2) . "%\n";
    echo "Selectivity: " . number_format($metrics->selectivity, 2) . "\n";
    echo "Days since last use: {$metrics->getDaysSinceLastUse()}\n";
}

// Get all usage metrics for a table
$allMetrics = $tracker->getTableUsageMetrics('users');

3. UnusedIndexDetector

Detects unused, duplicate, and redundant indexes.

Capabilities:

• Find unused indexes (configurable days threshold)
• Detect duplicate indexes (identical column coverage)
• Find redundant indexes (prefix patterns)
• Generate DROP statements for cleanup
• Estimate space savings

Usage:

use App\Framework\Database\Indexing\UnusedIndexDetector;

$detector = $container->get(UnusedIndexDetector::class);

// Find unused indexes (not used in last 30 days)
$unusedIndexes = $detector->findUnusedIndexes('users', daysThreshold: 30);

foreach ($unusedIndexes as $index) {
    echo "Unused: {$index['index_name']}\n";
    echo "Columns: " . implode(', ', $index['columns']) . "\n";
    echo "Last used: {$index['last_used_days_ago']} days ago\n";
    echo "Reason: {$index['reason']}\n";
}

// Find duplicate indexes
$duplicates = $detector->findDuplicateIndexes('users');

// Find redundant indexes (prefix pattern)
$redundant = $detector->findRedundantIndexes('users');

// Get comprehensive report
$report = $detector->getUnusedIndexReport('users', daysThreshold: 30);

echo "Total removable: {$report['total_removable']}\n";
echo "Estimated space savings: {$report['estimated_space_savings']}\n";

// Generate DROP statements
$dropStatements = $detector->generateDropStatements('users');

foreach ($dropStatements as $sql) {
    echo "{$sql}\n";
}

4. CompositeIndexGenerator

Generates smart composite index recommendations based on query patterns.

Capabilities:

• Analyze slow queries for index opportunities
• Suggest composite indexes (WHERE + ORDER BY columns)
• Detect full table scans needing indexes
• Estimate query speedup
• Prioritize recommendations (CRITICAL/HIGH/MEDIUM/LOW)

Usage:

use App\Framework\Database\Indexing\CompositeIndexGenerator;

$generator = $container->get(CompositeIndexGenerator::class);

// Generate recommendations for a table
$recommendations = $generator->generateRecommendations('users');

foreach ($recommendations as $recommendation) {
    echo "Priority: {$recommendation->priority->value}\n";
    echo "Index: {$recommendation->getIndexName()->toString()}\n";
    echo "Columns: {$recommendation->getColumnsString()}\n";
    echo "Reason: {$recommendation->reason}\n";
    echo "Estimated speedup: {$recommendation->estimatedSpeedup}x\n";
    echo "Affected queries: {$recommendation->affectedQueries}\n";
    echo "\n";
}

5. IndexMigrationGenerator

Generates database migration files for index optimizations.

Capabilities:

• Generate ADD INDEX migrations
• Generate DROP INDEX migrations
• Generate comprehensive optimization migrations (add + remove)
• Auto-save migrations with timestamp
• Include UP and DOWN methods for rollback

Usage:

use App\Framework\Database\Indexing\IndexMigrationGenerator;

$migrationGen = $container->get(IndexMigrationGenerator::class);

// Generate migration for adding recommended indexes
$recommendations = [/* IndexRecommendation objects */];
$migration = $migrationGen->generateAddIndexMigration($recommendations, 'users');

echo $migration; // PHP migration file content

// Generate migration for removing unused indexes
$unusedIndexes = [
    ['index_name' => 'idx_users_old', 'columns' => ['old_column']]
];
$migration = $migrationGen->generateRemoveIndexMigration($unusedIndexes, 'users');

// Generate comprehensive optimization migration
$migration = $migrationGen->generateOptimizationMigration(
    toAdd: $recommendations,
    toRemove: $unusedIndexes,
    tableName: 'users'
);

// Save migration to file
$path = $migrationGen->saveMigration($migration);
echo "Migration saved to: {$path}\n";

6. IndexOptimizationService

Facade service combining all index optimization components.

Capabilities:

• Complete table analysis (unused + recommendations)
• Generate optimization migrations automatically
• Index statistics dashboard
• High-priority recommendations across multiple tables
• Health check for optimization opportunities

Usage:

use App\Framework\Database\Indexing\IndexOptimizationService;

$service = $container->get(IndexOptimizationService::class);

// Complete table analysis
$analysis = $service->analyzeTable('users', unusedDaysThreshold: 30);

echo "Current indexes: " . count($analysis['current_indexes']) . "\n";
echo "Unused indexes: {$analysis['total_removable']}\n";
echo "Recommended indexes: {$analysis['total_recommended']}\n";
echo "Space savings: {$analysis['estimated_space_savings']}\n";

// Generate and save optimization migration
$migrationPath = $service->generateOptimizationMigration('users');
echo "Migration created: {$migrationPath}\n";

// Get index statistics
$stats = $service->getIndexStatistics('users');

// Get high-priority recommendations for multiple tables
$tables = ['users', 'orders', 'products'];
$highPriority = $service->getHighPriorityRecommendations($tables);

// Health check
$healthCheck = $service->healthCheck($tables, unusedDaysThreshold: 30);

if ($healthCheck['requires_attention']) {
    echo "⚠️  Optimization required:\n";
    echo "  - Tables with unused indexes: " .
         count($healthCheck['tables_with_unused_indexes']) . "\n";
    echo "  - Total removable: {$healthCheck['total_removable_indexes']}\n";
    echo "  - Total recommended: {$healthCheck['total_recommended_indexes']}\n";
}

Console Commands

Analyze Indexes

# Analyze specific table
php console.php db:analyze-indexes users

# Output:
# 🔍 Analyzing indexes for table: users
#
# 📊 Current Indexes (5 total):
#   - PRIMARY (PRIMARY): id
#   - idx_users_email (BTREE): email
#   - idx_users_status (BTREE): status
#   - idx_users_created_at (BTREE): created_at
#   - idx_users_email_status (BTREE): email, status
#
# 🗑️  Unused Indexes (2 total):
#   - idx_users_old_column: old_column (unused for 120 days)
#   - idx_users_deprecated: deprecated_field (unused for 90 days)
#
# 💡 Recommended Indexes (1 total):
#   - [HIGH] idx_users_status_created_at: status, created_at
#     Reason: WHERE status + ORDER BY created_at
#     Estimated speedup: 5.0x
#
# 📈 Summary:
#   - Removable indexes: 2
#   - Recommended indexes: 1
#   - Estimated space savings: 10 MB
#
# 💾 To generate migration, run:
#    php console.php db:generate-index-migration users

Value Objects

IndexName

Validated index name (1-64 characters, alphanumeric + underscore).

use App\Framework\Database\Indexing\ValueObjects\IndexName;

$indexName = new IndexName('idx_users_email');
echo $indexName->toString(); // "idx_users_email"

IndexType

Enum representing database index types.

use App\Framework\Database\Indexing\ValueObjects\IndexType;

$type = IndexType::BTREE;
echo $type->getDescription(); // "Balanced tree index - good for range queries"

// Database-specific support check
$isSupported = $type->isSupported('mysql'); // true

Supported Types:

• BTREE: Balanced tree (default, all databases)
• HASH: Hash index (MySQL, PostgreSQL)
• FULLTEXT: Full-text search (MySQL)
• SPATIAL: Geographic data (MySQL)
• GIN: Generalized Inverted Index (PostgreSQL)
• GIST: Generalized Search Tree (PostgreSQL)
• BRIN: Block Range Index (PostgreSQL)
• PRIMARY: Primary key
• UNIQUE: Unique constraint

IndexUsageMetrics

Statistics about index usage and effectiveness.

use App\Framework\Database\Indexing\ValueObjects\IndexUsageMetrics;

$metrics = new IndexUsageMetrics(
    indexName: new IndexName('idx_users_email'),
    tableName: 'users',
    usageCount: 15234,
    scanCount: 15234,
    selectivity: 0.95,
    rowsExamined: 152340,
    rowsReturned: 15234,
    lastUsed: new DateTimeImmutable('2025-01-19 14:30:00'),
    createdAt: new DateTimeImmutable('2024-12-01 10:00:00')
);

// Computed metrics
$efficiency = $metrics->getEfficiency(); // 0.10 (10% of examined rows returned)
$avgScanSize = $metrics->getAverageScanSize(); // 10 rows per scan
$daysSinceLastUse = $metrics->getDaysSinceLastUse(); // 0
$isUnused = $metrics->isUnused(daysThreshold: 30); // false

IndexRecommendation

Recommendation for creating or optimizing an index.

use App\Framework\Database\Indexing\ValueObjects\IndexRecommendation;
use App\Framework\Database\Indexing\ValueObjects\IndexType;
use App\Framework\Database\Indexing\ValueObjects\RecommendationPriority;

$recommendation = new IndexRecommendation(
    tableName: 'users',
    columns: ['status', 'created_at'],
    indexType: IndexType::BTREE,
    reason: 'WHERE status + ORDER BY created_at; Query uses filesort',
    priority: RecommendationPriority::HIGH,
    estimatedSpeedup: 5.0,
    affectedQueries: 100
);

$indexName = $recommendation->getIndexName(); // IndexName("idx_users_status_created_at")
$isComposite = $recommendation->isComposite(); // true
$array = $recommendation->toArray(); // Full array representation

RecommendationPriority

Priority levels for index recommendations.

use App\Framework\Database\Indexing\ValueObjects\RecommendationPriority;

// Auto-detect priority from metrics
$priority = RecommendationPriority::fromMetrics(
    speedup: 15.0,
    affectedQueries: 250
); // CRITICAL (>10x speedup or >100 affected queries)

// Priority levels:
// - CRITICAL: >10x speedup or >100 affected queries
// - HIGH: >5x speedup or >50 affected queries
// - MEDIUM: >2x speedup or >20 affected queries
// - LOW: <2x speedup or <20 affected queries

echo $priority->value; // "critical"
echo $priority->getColor(); // "red"

Best Practices

1. Regular Index Analysis

Run index analysis monthly or after major feature deployments:

# Analyze all critical tables
php console.php db:analyze-indexes users
php console.php db:analyze-indexes orders
php console.php db:analyze-indexes products

2. Unused Days Threshold

• Development: 7 days
• Staging: 14 days
• Production: 30-90 days (conservative)

3. Index Naming Convention

Generated index names follow pattern: idx_{table}_{column1}_{column2}

4. Composite Index Column Order

Rule: WHERE columns first, ORDER BY columns second

-- Query pattern:
SELECT * FROM users WHERE status = 'active' ORDER BY created_at DESC

-- Optimal index:
CREATE INDEX idx_users_status_created_at ON users(status, created_at)

5. Migration Safety

Always review generated migrations before running:

# Generate migration
php console.php db:generate-index-migration users

# Review file
cat migrations/20250119_optimize_indexes_for_users.php

# Apply migration
php console.php db:migrate

6. Monitor Index Effectiveness

Track index usage metrics regularly:

$metrics = $tracker->getTableUsageMetrics('users');

foreach ($metrics as $metric) {
    if ($metric->getEfficiency() < 0.1) {
        // Index returns <10% of examined rows - might need optimization
        $this->logger->warning("Low efficiency index", [
            'index' => $metric->indexName->toString(),
            'efficiency' => $metric->getEfficiency()
        ]);
    }
}

7. Avoid Over-Indexing

Costs of indexes:

• Storage space (5-10% of table size per index)
• INSERT/UPDATE/DELETE slowdown
• Maintenance overhead

Guidelines:

• Limit to 5-7 indexes per table
• Focus on high-traffic queries
• Remove unused indexes regularly

Integration with Existing Framework

ProfilingDashboard Integration

use App\Framework\Database\Profiling\ProfilingDashboard;
use App\Framework\Database\Indexing\IndexOptimizationService;

$dashboard = $container->get(ProfilingDashboard::class);
$indexService = $container->get(IndexOptimizationService::class);

// Generate comprehensive performance report
$report = $dashboard->generateReport();

// For each slow query, check if index would help
foreach ($report->slowQueries as $slowQuery) {
    $tableName = $this->extractTableName($slowQuery->query);
    $recommendations = $indexService->analyzeTable($tableName);

    if ($recommendations['total_recommended'] > 0) {
        echo "Index optimization available for {$tableName}\n";
    }
}

SlowQueryDetector Integration

use App\Framework\Database\Profiling\SlowQueryDetector;

$detector = $container->get(SlowQueryDetector::class);

// For N+1 query patterns, suggest composite indexes
$patterns = $detector->detectSlowQueryPatterns();

foreach ($patterns as $pattern) {
    if ($pattern->type === 'N_PLUS_ONE') {
        // Analyze and suggest composite index
        $recommendations = $generator->generateRecommendations($pattern->tableName);
    }
}

Performance Characteristics

Index Analysis:

• Typical Time: <100ms per table
• Memory Usage: <10MB for most tables
• Scalability: Linear with table size

Usage Tracking:

• Overhead: <1ms per query
• Cache Storage: ~1KB per index
• TTL: 30 days (configurable)

Migration Generation:

• Time: <50ms for typical table
• Output Size: 1-2KB per migration

Troubleshooting

Issue: "No recommendations found"

Cause: No slow queries or all queries already optimized Solution: Run queries for longer period to collect data

Issue: "Index marked as unused but is actually used"

Cause: Usage tracking not enabled or cache cleared Solution: Enable usage tracking for all queries:

// In query execution interceptor
$this->indexUsageTracker->recordUsage($sql, $tableName);

Issue: "Migration generation fails"

Cause: Invalid index name or table name Solution: Check IndexName validation (alphanumeric + underscore, max 64 chars)

Issue: "Duplicate index recommendations"

Cause: Similar query patterns analyzed multiple times Solution: Deduplication is automatic - review merged recommendations

Summary

The Index Optimization system provides:

• ✅ Automated index analysis with EXPLAIN parsing
• ✅ Usage tracking with cache-based metrics
• ✅ Unused index detection (unused/duplicate/redundant)
• ✅ Smart recommendations for composite indexes
• ✅ Automatic migration generation with rollback support
• ✅ Console commands for DBA workflows
• ✅ Framework integration with ProfilingDashboard and SlowQueryDetector
• ✅ Multi-database support (MySQL, PostgreSQL, SQLite)

Framework Compliance:

• Value Objects for type safety
• Readonly classes for immutability
• PSR-12 code style
• Comprehensive Pest tests
• Production-ready error handling