feat(Production): Complete production deployment infrastructure

- Add comprehensive health check system with multiple endpoints
- Add Prometheus metrics endpoint
- Add production logging configurations (5 strategies)
- Add complete deployment documentation suite:
  * QUICKSTART.md - 30-minute deployment guide
  * DEPLOYMENT_CHECKLIST.md - Printable verification checklist
  * DEPLOYMENT_WORKFLOW.md - Complete deployment lifecycle
  * PRODUCTION_DEPLOYMENT.md - Comprehensive technical reference
  * production-logging.md - Logging configuration guide
  * ANSIBLE_DEPLOYMENT.md - Infrastructure as Code automation
  * README.md - Navigation hub
  * DEPLOYMENT_SUMMARY.md - Executive summary
- Add deployment scripts and automation
- Add DEPLOYMENT_PLAN.md - Concrete plan for immediate deployment
- Update README with production-ready features

All production infrastructure is now complete and ready for deployment.
This commit is contained in:
2025-10-25 19:18:37 +02:00
parent caa85db796
commit fc3d7e6357
83016 changed files with 378904 additions and 20919 deletions

View File

@@ -0,0 +1,559 @@
# Database Index Optimization
Comprehensive guide for database index analysis and optimization in the Custom PHP Framework.
## Overview
The Index Optimization system provides automated tools for:
- **Index Usage Analysis**: Track real index usage statistics
- **Unused Index Detection**: Find indexes that waste storage and slow writes
- **Smart Recommendations**: Generate composite index suggestions based on query patterns
- **Automatic Migration Generation**: Create migration files for index optimizations
- **Performance Metrics**: Measure index effectiveness and query speedup
## Core Components
### 1. IndexAnalyzer
Core service for analyzing database index usage and effectiveness.
**Capabilities**:
- Parse EXPLAIN output (MySQL, PostgreSQL, SQLite)
- Detect actual index usage in queries
- Get all indexes for a table with metadata
- Multi-database support with driver-specific optimizations
**Usage**:
```php
use App\Framework\Database\Indexing\IndexAnalyzer;
$analyzer = $container->get(IndexAnalyzer::class);
// Get all indexes for a table
$indexes = $analyzer->getTableIndexes('users');
foreach ($indexes as $index) {
echo "Index: {$index['name']}\n";
echo "Columns: " . implode(', ', $index['columns']) . "\n";
echo "Type: {$index['type']->value}\n";
echo "Unique: " . ($index['is_unique'] ? 'Yes' : 'No') . "\n";
}
// Analyze query for index usage
$sql = 'SELECT * FROM users WHERE email = ? AND status = ?';
$analysis = $analyzer->analyzeQuery($sql);
echo "Indexes used: " . count($analysis['indexes_used']) . "\n";
echo "Key type: {$analysis['key_type']}\n";
echo "Rows examined: {$analysis['rows_examined']}\n";
echo "Using filesort: " . ($analysis['using_filesort'] ? 'Yes' : 'No') . "\n";
```
### 2. IndexUsageTracker
Tracks real index usage statistics over time using cache.
**Capabilities**:
- Record index usage for queries
- Calculate index selectivity and efficiency
- Track usage count and last used timestamp
- Generate usage metrics with Value Objects
**Usage**:
```php
use App\Framework\Database\Indexing\IndexUsageTracker;
use App\Framework\Database\Indexing\ValueObjects\IndexName;
$tracker = $container->get(IndexUsageTracker::class);
// Record usage for a query
$tracker->recordUsage('SELECT * FROM users WHERE email = ?', 'users');
// Get usage metrics for specific index
$indexName = new IndexName('idx_users_email');
$metrics = $tracker->getUsageMetrics($indexName, 'users');
if ($metrics) {
echo "Usage count: {$metrics->usageCount}\n";
echo "Efficiency: " . number_format($metrics->getEfficiency() * 100, 2) . "%\n";
echo "Selectivity: " . number_format($metrics->selectivity, 2) . "\n";
echo "Days since last use: {$metrics->getDaysSinceLastUse()}\n";
}
// Get all usage metrics for a table
$allMetrics = $tracker->getTableUsageMetrics('users');
```
### 3. UnusedIndexDetector
Detects unused, duplicate, and redundant indexes.
**Capabilities**:
- Find unused indexes (configurable days threshold)
- Detect duplicate indexes (identical column coverage)
- Find redundant indexes (prefix patterns)
- Generate DROP statements for cleanup
- Estimate space savings
**Usage**:
```php
use App\Framework\Database\Indexing\UnusedIndexDetector;
$detector = $container->get(UnusedIndexDetector::class);
// Find unused indexes (not used in last 30 days)
$unusedIndexes = $detector->findUnusedIndexes('users', daysThreshold: 30);
foreach ($unusedIndexes as $index) {
echo "Unused: {$index['index_name']}\n";
echo "Columns: " . implode(', ', $index['columns']) . "\n";
echo "Last used: {$index['last_used_days_ago']} days ago\n";
echo "Reason: {$index['reason']}\n";
}
// Find duplicate indexes
$duplicates = $detector->findDuplicateIndexes('users');
// Find redundant indexes (prefix pattern)
$redundant = $detector->findRedundantIndexes('users');
// Get comprehensive report
$report = $detector->getUnusedIndexReport('users', daysThreshold: 30);
echo "Total removable: {$report['total_removable']}\n";
echo "Estimated space savings: {$report['estimated_space_savings']}\n";
// Generate DROP statements
$dropStatements = $detector->generateDropStatements('users');
foreach ($dropStatements as $sql) {
echo "{$sql}\n";
}
```
### 4. CompositeIndexGenerator
Generates smart composite index recommendations based on query patterns.
**Capabilities**:
- Analyze slow queries for index opportunities
- Suggest composite indexes (WHERE + ORDER BY columns)
- Detect full table scans needing indexes
- Estimate query speedup
- Prioritize recommendations (CRITICAL/HIGH/MEDIUM/LOW)
**Usage**:
```php
use App\Framework\Database\Indexing\CompositeIndexGenerator;
$generator = $container->get(CompositeIndexGenerator::class);
// Generate recommendations for a table
$recommendations = $generator->generateRecommendations('users');
foreach ($recommendations as $recommendation) {
echo "Priority: {$recommendation->priority->value}\n";
echo "Index: {$recommendation->getIndexName()->toString()}\n";
echo "Columns: {$recommendation->getColumnsString()}\n";
echo "Reason: {$recommendation->reason}\n";
echo "Estimated speedup: {$recommendation->estimatedSpeedup}x\n";
echo "Affected queries: {$recommendation->affectedQueries}\n";
echo "\n";
}
```
### 5. IndexMigrationGenerator
Generates database migration files for index optimizations.
**Capabilities**:
- Generate ADD INDEX migrations
- Generate DROP INDEX migrations
- Generate comprehensive optimization migrations (add + remove)
- Auto-save migrations with timestamp
- Include UP and DOWN methods for rollback
**Usage**:
```php
use App\Framework\Database\Indexing\IndexMigrationGenerator;
$migrationGen = $container->get(IndexMigrationGenerator::class);
// Generate migration for adding recommended indexes
$recommendations = [/* IndexRecommendation objects */];
$migration = $migrationGen->generateAddIndexMigration($recommendations, 'users');
echo $migration; // PHP migration file content
// Generate migration for removing unused indexes
$unusedIndexes = [
['index_name' => 'idx_users_old', 'columns' => ['old_column']]
];
$migration = $migrationGen->generateRemoveIndexMigration($unusedIndexes, 'users');
// Generate comprehensive optimization migration
$migration = $migrationGen->generateOptimizationMigration(
toAdd: $recommendations,
toRemove: $unusedIndexes,
tableName: 'users'
);
// Save migration to file
$path = $migrationGen->saveMigration($migration);
echo "Migration saved to: {$path}\n";
```
### 6. IndexOptimizationService
Facade service combining all index optimization components.
**Capabilities**:
- Complete table analysis (unused + recommendations)
- Generate optimization migrations automatically
- Index statistics dashboard
- High-priority recommendations across multiple tables
- Health check for optimization opportunities
**Usage**:
```php
use App\Framework\Database\Indexing\IndexOptimizationService;
$service = $container->get(IndexOptimizationService::class);
// Complete table analysis
$analysis = $service->analyzeTable('users', unusedDaysThreshold: 30);
echo "Current indexes: " . count($analysis['current_indexes']) . "\n";
echo "Unused indexes: {$analysis['total_removable']}\n";
echo "Recommended indexes: {$analysis['total_recommended']}\n";
echo "Space savings: {$analysis['estimated_space_savings']}\n";
// Generate and save optimization migration
$migrationPath = $service->generateOptimizationMigration('users');
echo "Migration created: {$migrationPath}\n";
// Get index statistics
$stats = $service->getIndexStatistics('users');
// Get high-priority recommendations for multiple tables
$tables = ['users', 'orders', 'products'];
$highPriority = $service->getHighPriorityRecommendations($tables);
// Health check
$healthCheck = $service->healthCheck($tables, unusedDaysThreshold: 30);
if ($healthCheck['requires_attention']) {
echo "⚠️ Optimization required:\n";
echo " - Tables with unused indexes: " .
count($healthCheck['tables_with_unused_indexes']) . "\n";
echo " - Total removable: {$healthCheck['total_removable_indexes']}\n";
echo " - Total recommended: {$healthCheck['total_recommended_indexes']}\n";
}
```
## Console Commands
### Analyze Indexes
```bash
# Analyze specific table
php console.php db:analyze-indexes users
# Output:
# 🔍 Analyzing indexes for table: users
#
# 📊 Current Indexes (5 total):
# - PRIMARY (PRIMARY): id
# - idx_users_email (BTREE): email
# - idx_users_status (BTREE): status
# - idx_users_created_at (BTREE): created_at
# - idx_users_email_status (BTREE): email, status
#
# 🗑️ Unused Indexes (2 total):
# - idx_users_old_column: old_column (unused for 120 days)
# - idx_users_deprecated: deprecated_field (unused for 90 days)
#
# 💡 Recommended Indexes (1 total):
# - [HIGH] idx_users_status_created_at: status, created_at
# Reason: WHERE status + ORDER BY created_at
# Estimated speedup: 5.0x
#
# 📈 Summary:
# - Removable indexes: 2
# - Recommended indexes: 1
# - Estimated space savings: 10 MB
#
# 💾 To generate migration, run:
# php console.php db:generate-index-migration users
```
## Value Objects
### IndexName
Validated index name (1-64 characters, alphanumeric + underscore).
```php
use App\Framework\Database\Indexing\ValueObjects\IndexName;
$indexName = new IndexName('idx_users_email');
echo $indexName->toString(); // "idx_users_email"
```
### IndexType
Enum representing database index types.
```php
use App\Framework\Database\Indexing\ValueObjects\IndexType;
$type = IndexType::BTREE;
echo $type->getDescription(); // "Balanced tree index - good for range queries"
// Database-specific support check
$isSupported = $type->isSupported('mysql'); // true
```
**Supported Types**:
- `BTREE`: Balanced tree (default, all databases)
- `HASH`: Hash index (MySQL, PostgreSQL)
- `FULLTEXT`: Full-text search (MySQL)
- `SPATIAL`: Geographic data (MySQL)
- `GIN`: Generalized Inverted Index (PostgreSQL)
- `GIST`: Generalized Search Tree (PostgreSQL)
- `BRIN`: Block Range Index (PostgreSQL)
- `PRIMARY`: Primary key
- `UNIQUE`: Unique constraint
### IndexUsageMetrics
Statistics about index usage and effectiveness.
```php
use App\Framework\Database\Indexing\ValueObjects\IndexUsageMetrics;
$metrics = new IndexUsageMetrics(
indexName: new IndexName('idx_users_email'),
tableName: 'users',
usageCount: 15234,
scanCount: 15234,
selectivity: 0.95,
rowsExamined: 152340,
rowsReturned: 15234,
lastUsed: new DateTimeImmutable('2025-01-19 14:30:00'),
createdAt: new DateTimeImmutable('2024-12-01 10:00:00')
);
// Computed metrics
$efficiency = $metrics->getEfficiency(); // 0.10 (10% of examined rows returned)
$avgScanSize = $metrics->getAverageScanSize(); // 10 rows per scan
$daysSinceLastUse = $metrics->getDaysSinceLastUse(); // 0
$isUnused = $metrics->isUnused(daysThreshold: 30); // false
```
### IndexRecommendation
Recommendation for creating or optimizing an index.
```php
use App\Framework\Database\Indexing\ValueObjects\IndexRecommendation;
use App\Framework\Database\Indexing\ValueObjects\IndexType;
use App\Framework\Database\Indexing\ValueObjects\RecommendationPriority;
$recommendation = new IndexRecommendation(
tableName: 'users',
columns: ['status', 'created_at'],
indexType: IndexType::BTREE,
reason: 'WHERE status + ORDER BY created_at; Query uses filesort',
priority: RecommendationPriority::HIGH,
estimatedSpeedup: 5.0,
affectedQueries: 100
);
$indexName = $recommendation->getIndexName(); // IndexName("idx_users_status_created_at")
$isComposite = $recommendation->isComposite(); // true
$array = $recommendation->toArray(); // Full array representation
```
### RecommendationPriority
Priority levels for index recommendations.
```php
use App\Framework\Database\Indexing\ValueObjects\RecommendationPriority;
// Auto-detect priority from metrics
$priority = RecommendationPriority::fromMetrics(
speedup: 15.0,
affectedQueries: 250
); // CRITICAL (>10x speedup or >100 affected queries)
// Priority levels:
// - CRITICAL: >10x speedup or >100 affected queries
// - HIGH: >5x speedup or >50 affected queries
// - MEDIUM: >2x speedup or >20 affected queries
// - LOW: <2x speedup or <20 affected queries
echo $priority->value; // "critical"
echo $priority->getColor(); // "red"
```
## Best Practices
### 1. Regular Index Analysis
Run index analysis monthly or after major feature deployments:
```bash
# Analyze all critical tables
php console.php db:analyze-indexes users
php console.php db:analyze-indexes orders
php console.php db:analyze-indexes products
```
### 2. Unused Days Threshold
- **Development**: 7 days
- **Staging**: 14 days
- **Production**: 30-90 days (conservative)
### 3. Index Naming Convention
Generated index names follow pattern: `idx_{table}_{column1}_{column2}`
### 4. Composite Index Column Order
**Rule**: WHERE columns first, ORDER BY columns second
```sql
-- Query pattern:
SELECT * FROM users WHERE status = 'active' ORDER BY created_at DESC
-- Optimal index:
CREATE INDEX idx_users_status_created_at ON users(status, created_at)
```
### 5. Migration Safety
Always review generated migrations before running:
```bash
# Generate migration
php console.php db:generate-index-migration users
# Review file
cat migrations/20250119_optimize_indexes_for_users.php
# Apply migration
php console.php db:migrate
```
### 6. Monitor Index Effectiveness
Track index usage metrics regularly:
```php
$metrics = $tracker->getTableUsageMetrics('users');
foreach ($metrics as $metric) {
if ($metric->getEfficiency() < 0.1) {
// Index returns <10% of examined rows - might need optimization
$this->logger->warning("Low efficiency index", [
'index' => $metric->indexName->toString(),
'efficiency' => $metric->getEfficiency()
]);
}
}
```
### 7. Avoid Over-Indexing
**Costs of indexes**:
- Storage space (5-10% of table size per index)
- INSERT/UPDATE/DELETE slowdown
- Maintenance overhead
**Guidelines**:
- Limit to 5-7 indexes per table
- Focus on high-traffic queries
- Remove unused indexes regularly
## Integration with Existing Framework
### ProfilingDashboard Integration
```php
use App\Framework\Database\Profiling\ProfilingDashboard;
use App\Framework\Database\Indexing\IndexOptimizationService;
$dashboard = $container->get(ProfilingDashboard::class);
$indexService = $container->get(IndexOptimizationService::class);
// Generate comprehensive performance report
$report = $dashboard->generateReport();
// For each slow query, check if index would help
foreach ($report->slowQueries as $slowQuery) {
$tableName = $this->extractTableName($slowQuery->query);
$recommendations = $indexService->analyzeTable($tableName);
if ($recommendations['total_recommended'] > 0) {
echo "Index optimization available for {$tableName}\n";
}
}
```
### SlowQueryDetector Integration
```php
use App\Framework\Database\Profiling\SlowQueryDetector;
$detector = $container->get(SlowQueryDetector::class);
// For N+1 query patterns, suggest composite indexes
$patterns = $detector->detectSlowQueryPatterns();
foreach ($patterns as $pattern) {
if ($pattern->type === 'N_PLUS_ONE') {
// Analyze and suggest composite index
$recommendations = $generator->generateRecommendations($pattern->tableName);
}
}
```
## Performance Characteristics
**Index Analysis**:
- **Typical Time**: <100ms per table
- **Memory Usage**: <10MB for most tables
- **Scalability**: Linear with table size
**Usage Tracking**:
- **Overhead**: <1ms per query
- **Cache Storage**: ~1KB per index
- **TTL**: 30 days (configurable)
**Migration Generation**:
- **Time**: <50ms for typical table
- **Output Size**: 1-2KB per migration
## Troubleshooting
### Issue: "No recommendations found"
**Cause**: No slow queries or all queries already optimized
**Solution**: Run queries for longer period to collect data
### Issue: "Index marked as unused but is actually used"
**Cause**: Usage tracking not enabled or cache cleared
**Solution**: Enable usage tracking for all queries:
```php
// In query execution interceptor
$this->indexUsageTracker->recordUsage($sql, $tableName);
```
### Issue: "Migration generation fails"
**Cause**: Invalid index name or table name
**Solution**: Check IndexName validation (alphanumeric + underscore, max 64 chars)
### Issue: "Duplicate index recommendations"
**Cause**: Similar query patterns analyzed multiple times
**Solution**: Deduplication is automatic - review merged recommendations
## Summary
The Index Optimization system provides:
-**Automated index analysis** with EXPLAIN parsing
-**Usage tracking** with cache-based metrics
-**Unused index detection** (unused/duplicate/redundant)
-**Smart recommendations** for composite indexes
-**Automatic migration generation** with rollback support
-**Console commands** for DBA workflows
-**Framework integration** with ProfilingDashboard and SlowQueryDetector
-**Multi-database support** (MySQL, PostgreSQL, SQLite)
**Framework Compliance**:
- Value Objects for type safety
- Readonly classes for immutability
- PSR-12 code style
- Comprehensive Pest tests
- Production-ready error handling