fix(Discovery): Add comprehensive debug logging for router initialization
- Add initializer count logging in DiscoveryServiceBootstrapper - Add route structure analysis in RouterSetup - Add request parameter logging in HttpRouter - Update PHP production config for better OPcache handling - Fix various config and error handling improvements
This commit is contained in:
286
deployment/infrastructure/DEPLOYMENT_FIX_SUMMARY.md
Normal file
286
deployment/infrastructure/DEPLOYMENT_FIX_SUMMARY.md
Normal file
@@ -0,0 +1,286 @@
|
||||
# Production Deployment Fix Summary
|
||||
|
||||
**Date**: 2025-10-27
|
||||
**Status**: PARTIALLY FIXED - DB configuration corrected, but additional issues remain
|
||||
|
||||
---
|
||||
|
||||
## What Was Fixed
|
||||
|
||||
### 1. Database Configuration Corrected ✅
|
||||
|
||||
**Problem**: Wrong DB_PORT in production `.env.production`
|
||||
- Line 15: `DB_PORT=3306` (MySQL port)
|
||||
- Line 67: `DB_PORT=` (duplicate empty entry)
|
||||
- Wrong username: `mdb-user` (should be `mdb_user`)
|
||||
- Wrong password
|
||||
|
||||
**Solution Applied**:
|
||||
```bash
|
||||
# Copied correct .env.production from source of truth
|
||||
ansible web_servers -m copy \
|
||||
-a "src=deployment/applications/environments/.env.production \
|
||||
dest=/home/deploy/michaelschiemer/shared/.env.production" \
|
||||
--vault-password-file deployment/infrastructure/.vault_pass
|
||||
```
|
||||
|
||||
**Verification**:
|
||||
```bash
|
||||
DB_PORT=5432 # ✅ Correct
|
||||
DB_USERNAME=mdb_user # ✅ Correct
|
||||
DB_PASSWORD=Qo2KNgGqeYksEhKr57pgugakxlothn8J # ✅ Correct
|
||||
```
|
||||
|
||||
### 2. Containers Restarted ✅
|
||||
|
||||
```bash
|
||||
docker compose restart php web queue-worker
|
||||
```
|
||||
|
||||
**Current Status**:
|
||||
- **php**: Up 6 minutes (healthy) ✅
|
||||
- **db**: Up 53 minutes (healthy) ✅
|
||||
- **redis**: Up 53 minutes (healthy) ✅
|
||||
- **web**: Up 6 minutes (UNHEALTHY) ⚠️
|
||||
- **queue-worker**: Restarting (1) ❌
|
||||
|
||||
---
|
||||
|
||||
## Remaining Issues
|
||||
|
||||
### Issue 1: Web Container Unhealthy ⚠️
|
||||
|
||||
**Symptom**: Website still returns HTTP 500
|
||||
|
||||
**Possible Causes**:
|
||||
1. **PHP-FPM not responding** - Web container can't connect to PHP
|
||||
2. **Application error** - PHP code failing during bootstrap
|
||||
3. **Missing files** - Application files not properly deployed
|
||||
4. **Permissions** - Web server can't access application files
|
||||
|
||||
**Next Steps to Diagnose**:
|
||||
```bash
|
||||
# Check if PHP-FPM is accessible from web container
|
||||
docker exec web curl http://php:9000
|
||||
|
||||
# Check Nginx configuration
|
||||
docker exec web nginx -t
|
||||
|
||||
# Check web container health check
|
||||
docker inspect web --format='{{json .State.Health}}' | jq
|
||||
|
||||
# Check if application files exist
|
||||
docker exec web ls -la /var/www/html/public/index.php
|
||||
```
|
||||
|
||||
### Issue 2: Queue Worker Crashing ❌
|
||||
|
||||
**Symptom**: Continuous restart loop
|
||||
|
||||
**Possible Causes**:
|
||||
1. **Same DB connection issue** (should be fixed now)
|
||||
2. **Missing queue configuration**
|
||||
3. **Redis connection issue**
|
||||
4. **Application code error in queue worker**
|
||||
|
||||
**Next Steps to Diagnose**:
|
||||
```bash
|
||||
# Check queue-worker logs
|
||||
docker logs queue-worker --tail 100
|
||||
|
||||
# Try running queue worker manually
|
||||
docker exec php php artisan queue:work --tries=1 --once
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Scripts Created ✅
|
||||
|
||||
### 1. Simple Deployment Script
|
||||
**Location**: `/home/michael/dev/michaelschiemer/deployment/infrastructure/scripts/deploy.sh`
|
||||
```bash
|
||||
./deployment/infrastructure/scripts/deploy.sh
|
||||
```
|
||||
|
||||
### 2. .env Update Script
|
||||
**Location**: `/home/michael/dev/michaelschiemer/deployment/infrastructure/scripts/update-env.sh`
|
||||
```bash
|
||||
./deployment/infrastructure/scripts/update-env.sh
|
||||
```
|
||||
|
||||
### 3. Quick Sync Script
|
||||
**Location**: `/home/michael/dev/michaelschiemer/deployment/infrastructure/scripts/quick-sync.sh`
|
||||
```bash
|
||||
./deployment/infrastructure/scripts/quick-sync.sh
|
||||
```
|
||||
|
||||
**Note**: All scripts updated to use `docker compose` (v2) instead of `docker-compose` (v1)
|
||||
|
||||
---
|
||||
|
||||
## Documentation Created ✅
|
||||
|
||||
### Comprehensive Deployment Analysis
|
||||
**Location**: `/home/michael/dev/michaelschiemer/deployment/infrastructure/DEPLOYMENT_ANALYSIS.md`
|
||||
|
||||
**Contents**:
|
||||
1. Complete deployment flow analysis
|
||||
2. .env file sources and conflicts
|
||||
3. Deployment command documentation
|
||||
4. Step-by-step fix strategy
|
||||
5. Cleanup recommendations
|
||||
6. Post-fix verification checklist
|
||||
|
||||
---
|
||||
|
||||
## Recommended Next Actions
|
||||
|
||||
### Immediate (To Fix HTTP 500)
|
||||
|
||||
1. **Check Application Bootstrap**:
|
||||
```bash
|
||||
# Test if PHP application can start
|
||||
ansible web_servers -m shell \
|
||||
-a "docker exec php php /var/www/html/public/index.php" \
|
||||
--vault-password-file deployment/infrastructure/.vault_pass
|
||||
```
|
||||
|
||||
2. **Check Nginx-PHP Connection**:
|
||||
```bash
|
||||
# Test PHP-FPM socket
|
||||
ansible web_servers -m shell \
|
||||
-a "docker exec web curl -v http://php:9000" \
|
||||
--vault-password-file deployment/infrastructure/.vault_pass
|
||||
```
|
||||
|
||||
3. **Check Application Logs**:
|
||||
```bash
|
||||
# Look for PHP errors
|
||||
ansible web_servers -m shell \
|
||||
-a "docker exec php ls -la /var/www/html/storage/logs/" \
|
||||
--vault-password-file deployment/infrastructure/.vault_pass
|
||||
```
|
||||
|
||||
4. **Verify File Permissions**:
|
||||
```bash
|
||||
# Check if web server can read files
|
||||
ansible web_servers -m shell \
|
||||
-a "docker exec web ls -la /var/www/html/public/" \
|
||||
--vault-password-file deployment/infrastructure/.vault_pass
|
||||
```
|
||||
|
||||
### Short-Term (Within 24h)
|
||||
|
||||
1. **Fix Web Container Health** - Resolve HTTP 500 errors
|
||||
2. **Fix Queue Worker** - Stop crash loop
|
||||
3. **Full Deployment Test** - Run complete deployment playbook
|
||||
4. **Verify All Services** - Ensure all containers healthy
|
||||
|
||||
### Long-Term (This Week)
|
||||
|
||||
1. **Update Playbook** - Add .env.production sync task
|
||||
2. **Add Validation** - Pre-deployment .env validation script
|
||||
3. **Document Process** - Update README with deployment guide
|
||||
4. **Setup Monitoring** - Add health check alerts
|
||||
5. **Cleanup Old Files** - Remove duplicate .env files
|
||||
|
||||
---
|
||||
|
||||
## Key Learnings
|
||||
|
||||
### 1. Deployment Flow Issues
|
||||
|
||||
**Problem**: Playbook doesn't sync `.env.production` to `shared/`
|
||||
**Impact**: Manual updates required for configuration changes
|
||||
**Solution**: Add sync task to playbook
|
||||
|
||||
### 2. Multiple .env Sources
|
||||
|
||||
**Problem**: 3 different `.env.production` files with conflicting content
|
||||
**Resolution**: Use `deployment/applications/environments/.env.production` as source of truth
|
||||
|
||||
### 3. Docker Compose Version
|
||||
|
||||
**Problem**: Production uses Docker Compose v2 (`docker compose`)
|
||||
**Impact**: Scripts using v1 syntax (`docker-compose`) fail
|
||||
**Solution**: All scripts updated to v2 syntax
|
||||
|
||||
### 4. Symlink Chain Complexity
|
||||
|
||||
**Structure**:
|
||||
```
|
||||
current/.env → shared/.env.production
|
||||
current/.env.production → shared/.env.production
|
||||
```
|
||||
|
||||
**Risk**: If `shared/.env.production` is wrong, ALL releases break
|
||||
**Mitigation**: Validate before deploy, backup before changes
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Check Production Status
|
||||
```bash
|
||||
cd /home/michael/dev/michaelschiemer/deployment/infrastructure
|
||||
|
||||
# Container status
|
||||
ansible web_servers -i inventories/production/hosts.yml \
|
||||
-m shell -a "docker ps" --vault-password-file .vault_pass
|
||||
|
||||
# .env configuration
|
||||
ansible web_servers -i inventories/production/hosts.yml \
|
||||
-m shell -a "cat /home/deploy/michaelschiemer/shared/.env.production" \
|
||||
--vault-password-file .vault_pass
|
||||
|
||||
# Application logs
|
||||
ansible web_servers -i inventories/production/hosts.yml \
|
||||
-m shell -a "docker logs web --tail 50" --vault-password-file .vault_pass
|
||||
```
|
||||
|
||||
### Deploy to Production
|
||||
```bash
|
||||
# Full deployment
|
||||
./deployment/infrastructure/scripts/deploy.sh
|
||||
|
||||
# Update .env only
|
||||
./deployment/infrastructure/scripts/update-env.sh
|
||||
|
||||
# Quick code sync
|
||||
./deployment/infrastructure/scripts/quick-sync.sh
|
||||
```
|
||||
|
||||
### Emergency Rollback
|
||||
```bash
|
||||
# List releases
|
||||
ansible web_servers -i inventories/production/hosts.yml \
|
||||
-m shell -a "ls -la /home/deploy/michaelschiemer/releases/" \
|
||||
--vault-password-file .vault_pass
|
||||
|
||||
# Switch to previous release
|
||||
ansible web_servers -i inventories/production/hosts.yml \
|
||||
-m shell -a "ln -sfn /home/deploy/michaelschiemer/releases/PREVIOUS_TIMESTAMP \
|
||||
/home/deploy/michaelschiemer/current" \
|
||||
--vault-password-file .vault_pass
|
||||
|
||||
# Restart containers
|
||||
ansible web_servers -i inventories/production/hosts.yml \
|
||||
-m shell -a "cd /home/deploy/michaelschiemer/current && docker compose restart" \
|
||||
--vault-password-file .vault_pass
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Support Contacts
|
||||
|
||||
**Documentation**:
|
||||
- Deployment Analysis: `deployment/infrastructure/DEPLOYMENT_ANALYSIS.md`
|
||||
- This Summary: `deployment/infrastructure/DEPLOYMENT_FIX_SUMMARY.md`
|
||||
|
||||
**Scripts**:
|
||||
- All scripts in: `deployment/infrastructure/scripts/`
|
||||
- Make executable: `chmod +x deployment/infrastructure/scripts/*.sh`
|
||||
|
||||
**Configuration**:
|
||||
- Source of Truth: `deployment/applications/environments/.env.production`
|
||||
- Production File: `/home/deploy/michaelschiemer/shared/.env.production`
|
||||
Reference in New Issue
Block a user