michael/michaelschiemer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: DockerSecretsResolver - don't normalize absolute paths like /var/www/html/...
Some checks failed
Deploy Application / deploy (push) Has been cancelled
Details

Browse Source
This commit is contained in:
Michael Schiemer
2025-11-24 21:28:25 +01:00
parent 4eb7134853
commit 77abc65cd7
1327 changed files with 91915 additions and 9909 deletions
Download Patch File Download Diff File Expand all files Collapse all files

538
deployment/README.md
View File

@@ -1,17 +1,48 @@
# Pragmatic Production Deployment Setup
# Two-Layer Deployment System
## Architecture Overview
This deployment setup uses separate Docker Compose stacks for better maintainability and clear separation of concerns.
Das Deployment-System verwendet eine klare Trennung zwischen Infrastruktur-Layer und Application-Layer:
- **Layer 1 (Infrastruktur)**: Traefik, Gitea, PostgreSQL - separat deployt, läuft dauerhaft
- **Layer 2 (Application)**: PHP-App mit Nginx, Redis, Queue Workers - deployt aus dem Projekt
### Vorteile
- ✅ Klare Trennung: Infrastruktur vs. Application
- ✅ Einfach zu verwalten: Jede Schicht separat verwaltbar
- ✅ Gitea separat: Unabhängige Updates möglich
- ✅ Nutzt bestehende Struktur: Base+Override Pattern bleibt erhalten
- ✅ Skalierbar: Einfach erweiterbar
## Infrastructure Layer
Die Infrastruktur-Stacks befinden sich in `deployment/infrastructure/`:
- **Traefik** - Reverse Proxy mit SSL-Zertifikaten
- **Gitea** - Git Server mit eigener PostgreSQL-Instanz
- **PostgreSQL** - Shared Database für Application-Stacks
**Dokumentation:** Siehe [deployment/infrastructure/README.md](infrastructure/README.md)
**Deployment:**
```bash
cd deployment/infrastructure
./deploy.sh all # Deploys alle Stacks in korrekter Reihenfolge
```
## Application Layer
Die Application wird aus dem Projekt deployt und nutzt externe Infrastruktur über Docker Networks.
### Docker Compose Structure
The project uses a **Base + Override Pattern** to prevent configuration drift between environments:
Das Projekt verwendet ein **Base + Override Pattern**:
- **`docker-compose.base.yml`** - Shared base configuration (services, networks, volumes)
- **`docker-compose.local.yml`** - Local development overrides (ports, host mounts, debug flags)
- **`docker-compose.staging.yml`** - Staging environment overrides (Traefik labels, staging volumes)
- **`docker-compose.production.yml`** - Production environment overrides (security, logging, resources)
- **`docker-compose.base.yml`** - Gemeinsame Basis-Konfiguration
- **`docker-compose.local.yml`** - Local Development Overrides
- **`docker-compose.staging.yml`** - Staging Environment Overrides
- **`docker-compose.prod.yml`** - Production Environment Overrides
**Usage:**
```bash
@@ -22,373 +53,186 @@ docker compose -f docker-compose.base.yml -f docker-compose.local.yml up
docker compose -f docker-compose.base.yml -f docker-compose.staging.yml up
# Production
docker compose -f docker-compose.base.yml -f docker-compose.production.yml up
docker compose -f docker-compose.base.yml -f docker-compose.prod.yml up
```
**Benefits:**
- ✅ Single source of truth for shared configuration
- ✅ Environment-specific differences clearly visible
- ✅ Reduced configuration drift between environments
- ✅ Easier maintenance and updates
## Deployment Workflow
### Infrastructure Components
### Automatisches Deployment (Gitea Actions)
```
Production Server (94.16.110.151)
├── Stack 1: Traefik (Reverse Proxy & SSL)
├── Stack 2: Gitea (Git Server + MySQL + Redis)
├── Stack 3: Docker Registry (Private Registry)
├── Stack 4: Application (PHP + Nginx + Redis + Queue Workers)
├── Stack 5: PostgreSQL (Database)
└── Stack 6: Monitoring (Portainer + Grafana + Prometheus)
**Workflow:** `.gitea/workflows/deploy.yml`
Development Machine
└── Gitea Actions Runner (local, Docker-in-Docker)
- Trigger: Push zu `staging` oder `main` Branch
- Führt automatisch Deployment-Script aus
- Status-Reporting zurück zu Gitea
### Manuelles Deployment (SSH-Script)
**Script:** `deployment/scripts/deploy.sh`
```bash
# Staging deployen
./deployment/scripts/deploy.sh staging
# Production deployen
./deployment/scripts/deploy.sh production
# Mit Image-Build
./deployment/scripts/deploy.sh staging build
```
## Deployment Flow
**Was passiert:**
1. Secrets-Prüfung
2. Infrastructure-Networks-Prüfung
3. Docker Images pullen (optional: builden)
4. Docker Compose Up
5. Health Checks
6. Status-Report
```
Developer → git push
↓
Gitea (Production)
↓
Gitea Actions (Dev Machine)
↓
Build Docker Image
↓
Push to Private Registry
↓
SSH/Ansible → Production Server
↓
docker compose pull
↓
docker compose up -d
```
## Networks
## Directory Structure
Das System verwendet folgende Docker Networks:
### Local Repository Structure
- **traefik-public** - Wird von Traefik erstellt, für externe Zugriffe
- **infrastructure** - Für interne Infrastruktur-Kommunikation (Gitea ↔ PostgreSQL)
- **app-internal** - Wird von PostgreSQL erstellt, für Application ↔ PostgreSQL Kommunikation
- **app-backend** - Internes Network für Application-Services (PHP ↔ Nginx ↔ Redis)
## Secrets Management
Secrets werden in `deployment/secrets/` Verzeichnissen gespeichert:
```
deployment/
├── ansible/ # Ansible config, playbooks, inventory, templates
├── gitea-runner/ # Self-hosted Gitea Actions runner stack
├── stacks/ # Docker Compose stacks
│ ├── application/ # Main PHP application
│ ├── gitea/ # Git server
│ ├── minio/ # Object storage
│ ├── monitoring/ # Portainer, Grafana, Prometheus
│ ├── postgresql/ # PostgreSQL database
│ ├── registry/ # Private Docker registry
│ ├── staging/ # Optional staging stack
│ └── traefik/ # Reverse proxy with SSL certificates
├── docs/ # 📚 Dokumentation (siehe docs/README.md)
│ ├── guides/ # Anleitungen & Guides
│ ├── reference/ # Referenz-Dokumentation
│ ├── status/ # Status & Tracking
│ ├── tests/ # Test-Dokumentation
│ └── history/ # Logs & Historie
├── infrastructure/
│ ├── traefik/secrets/
│ ├── gitea/secrets/
│ └── postgresql/secrets/
└── secrets/
├── staging/
│ ├── db_password.txt
│ ├── redis_password.txt
│ └── app_key.txt
└── production/
├── db_password.txt
├── redis_password.txt
└── app_key.txt
```
**Wichtig:** Secrets-Dateien sind gitignored und müssen manuell erstellt werden.
Siehe [deployment/infrastructure/SECRETS.md](infrastructure/SECRETS.md) für Details.
## Quick Start
### Initial Setup (einmalig)
1. **Infrastruktur deployen:**
```bash
cd deployment/infrastructure
./deploy.sh all
```
2. **Secrets konfigurieren:**
```bash
# Siehe deployment/infrastructure/SECRETS.md
```
3. **Application deployen:**
```bash
./deployment/scripts/deploy.sh staging
```
### Normaler Deployment-Workflow
1. **Code ändern und committen:**
```bash
git add .
git commit -m "feat: Add new feature"
git push origin staging # → Automatisches Deployment zu Staging
```
2. **Testen auf Staging:**
- Staging URL: `https://staging.michaelschiemer.de`
- Tests durchführen
3. **Nach erfolgreichem Test zu Production:**
```bash
git checkout main
git merge staging
git push origin main # → Automatisches Deployment zu Production
```
## Migration vom alten System
Falls Sie vom alten System migrieren, siehe [MIGRATION.md](MIGRATION.md) für eine detaillierte Anleitung.
## Directory Structure
```
deployment/
├── infrastructure/ # Infrastructure Layer
│ ├── traefik/
│ ├── gitea/
│ ├── postgresql/
│ ├── deploy.sh
│ └── README.md
├── scripts/ # Deployment Scripts
│ └── deploy.sh
├── secrets/ # Application Secrets (gitignored)
│ ├── staging/
│ └── production/
├── legacy/ # Altes System (nur Referenz)
└── README.md (dieses Dokument)
```
### Server Directory Structure
Auf dem Production-Server existieren zwei Hauptverzeichnisse:
```
/home/deploy/
├── deployment/ # Infrastructure-as-Code (24M)
│ ├── stacks/ # Docker Compose Stacks
│ └── backups/ # Backup-Dateien
└── michaelschiemer/ # Application Code (491M)
├── current/ # Symlink → Aktuelle deployed Version
└── .archive/ # Alte Versionen (Rollback)
```
**📖 Detaillierte Erklärung:** Siehe [docs/server-directory-structure.md](docs/server-directory-structure.md)
## Getting Started
### 🧪 Pipeline-Tests vorbereiten
**Vor dem ersten Deployment:**
1. **Prerequisites prüfen:**
```bash
./deployment/scripts/test-pipeline-prerequisites.sh
```
2. **Test-Anleitung lesen:**
- [Pipeline Test Checklist](docs/guides/pipeline-test-checklist.md) ⭐ - Schritt-für-Schritt Anleitung
- [Pipeline Testing Guide](docs/guides/pipeline-testing-guide.md) - Übersicht und Troubleshooting
3. **Backup-Test durchführen:**
```bash
./deployment/scripts/test-backup.sh
```
### 🚀 Quick Start: Code deployen
**Empfohlener Workflow (Staging → Production):**
1. **Push auf `staging` Branch** (Standard für Entwicklung)
```bash
git add .
git commit -m "feat: Add new feature"
git push origin staging # → Automatisches Deployment zu Staging
```
2. **Testen auf Staging**
- Staging URL: `https://staging.michaelschiemer.de`
- Tests durchführen und verifizieren
3. **Merge nach `main`** (nur nach erfolgreichem Test)
```bash
git checkout main
git merge staging
git push origin main # → Automatisches Deployment zu Production
```
**⚠️ Wichtig:** Niemals direkt auf `main` pushen - immer erst auf `staging` testen!
**Pipeline-Status:** `https://git.michaelschiemer.de/michael/michaelschiemer/actions`
**📖 Vollständige Anleitung:** Siehe [docs/guides/quick-start.md](docs/guides/quick-start.md) oder [docs/guides/code-change-workflow.md](docs/guides/code-change-workflow.md)
---
### Initial Setup (nur bei erstem Setup)
**Prerequisites:**
**Production Server:**
- Docker & Docker Compose installed
- Firewall configured (ports 80, 443, 2222)
- User `deploy` with Docker permissions
- SSH access configured
**Development Machine:**
- Docker & Docker Compose installed
- Ansible installed
- SSH key configured for production server
**Deployment via Ansible:**
```bash
cd deployment/ansible
ansible-playbook -i inventory/production.yml playbooks/setup-infrastructure.yml
```
Dieses Playbook deployed alle Stacks:
- Traefik (Reverse Proxy & SSL)
- PostgreSQL (Database)
- Docker Registry (Private Registry)
- Gitea (Git Server)
- Monitoring (Portainer, Grafana, Prometheus)
- **Production Stack** (PHP Application + Nginx + Redis + Queue Workers)
**Gitea Initial Setup (nach Infrastructure Deployment):**
```bash
# Automatische Initial Setup via Ansible
cd deployment/ansible
# 1. Gitea Initial Configuration (Admin-User erstellen)
ansible-playbook -i inventory/production.yml \
playbooks/setup-gitea-initial-config.yml \
--vault-password-file secrets/.vault_pass
# 2. Repository in Gitea erstellen und Git-Remote konfigurieren
ansible-playbook -i inventory/production.yml \
playbooks/setup-gitea-repository.yml \
--vault-password-file secrets/.vault_pass \
-e "repo_name=michaelschiemer" \
-e "repo_owner=michael" \
-e "repo_private=false"
```
**📖 Vollständige Setup-Anleitung:** Siehe [SETUP-GUIDE.md](SETUP-GUIDE.md)
## Stack Documentation
Each stack has its own README with detailed configuration:
- [Traefik](stacks/traefik/README.md) - Reverse proxy setup
- [Gitea](stacks/gitea/README.md) - Git server configuration
- [Registry](stacks/registry/README.md) - Private registry setup
- [Production](stacks/production/README.md) - Production application deployment
- [PostgreSQL](stacks/postgresql/README.md) - Database configuration
- [Monitoring](stacks/monitoring/README.md) - Monitoring stack
## Deployment Commands
### Automatisches Deployment (Empfohlen)
**Standard-Workflow: Staging → Production**
1. **Push auf `staging`** (Standard für Entwicklung)
```bash
git add .
git commit -m "feat: Add new feature"
git push origin staging # → Deployt zu Staging
```
2. **Testen auf Staging**, dann **Merge nach `main`**
```bash
git checkout main
git merge staging
git push origin main # → Deployt zu Production
```
**📖 Vollständige Command-Referenz:** Siehe [docs/guides/deployment-commands.md](docs/guides/deployment-commands.md)
### Update Specific Stack
```bash
cd stacks/<stack-name>
docker compose pull
docker compose up -d
```
## CI/CD Pipeline
The CI/CD pipeline is defined in `.gitea/workflows/build-image.yml` and runs automatically on push to `staging` or `main` branch.
### Recommended Workflow: Staging → Production
**1. Push to `staging` (Standard for Development)**
```bash
# Make changes locally
# ... edit files ...
# Commit and push to staging
git add .
git commit -m "feat: Add new feature"
git push origin staging # → Deploys to Staging
```
**What happens automatically on `staging`:**
- ✅ Tests run (~2-5 min)
- ✅ Docker image is built (~3-5 min)
- ✅ Image is pushed to registry (~1-2 min)
- ✅ Deployment to Staging via SSH/SCP (~2-4 min)
- ✅ Staging stack is updated
**2. Test on Staging**
- Staging URL: `https://staging.michaelschiemer.de`
- Verify functionality and run tests
**3. Merge to `main` (Only after successful testing)**
```bash
git checkout main
git merge staging
git push origin main # → Deploys to Production
```
**What happens automatically on `main`:**
- ✅ Tests run (~2-5 min)
- ✅ Docker image is built (~3-5 min)
- ✅ Image is pushed to registry (~1-2 min)
- ✅ Deployment to Production via SSH/SCP (~2-4 min)
- ✅ Production stack is updated
**Total time per deployment:** ~8-15 minutes
**Status check:**
- Pipeline status: `https://git.michaelschiemer.de/michael/michaelschiemer/actions`
- Staging status: `ssh deploy@94.16.110.151 "cd ~/deployment/stacks/staging && docker compose ps"`
- Production status: `ssh deploy@94.16.110.151 "cd ~/deployment/stacks/production && docker compose ps"`
**⚠️ Important:** Never push directly to `main` - always test on `staging` first!
**📖 Vollständige Dokumentation:**
- **[docs/guides/quick-start.md](docs/guides/quick-start.md)** ⭐ - Schnellstart-Guide für Deployment
- **[docs/guides/code-change-workflow.md](docs/guides/code-change-workflow.md)** - Kompletter Guide für Codeänderungen
- **[docs/reference/application-stack.md](docs/reference/application-stack.md)** - Detaillierter Deployment-Ablauf
- **[docs/status/ci-cd-status.md](docs/status/ci-cd-status.md)** - CI/CD Pipeline Status & Checkliste
- **[docs/status/deployment-summary.md](docs/status/deployment-summary.md)** - Projekt-Status Übersicht
### Pipeline Details
The CI/CD pipeline runs on push to `staging` or `main` branch:
**On `staging` branch:**
1. **Build Stage**: Build Docker image
2. **Push Stage**: Push to private registry
3. **Deploy Stage**: Deploy to Staging via SSH/SCP
**On `main` branch:**
1. **Build Stage**: Build Docker image
2. **Push Stage**: Push to private registry
3. **Deploy Stage**: Deploy to Production via SSH/SCP
## Monitoring
Access monitoring tools:
- **Portainer**: https://portainer.yourdomain.com
- **Grafana**: https://grafana.yourdomain.com
- **Prometheus**: https://prometheus.yourdomain.com
## Backup & Recovery
### Current State
Infrastructure backups are handled per stack. The PostgreSQL stack ships helper scripts under `stacks/postgresql/scripts/` (see `backup.sh` and `restore.sh`). Registry and Gitea data snapshots are currently managed manually on the host.
### Roadmap
An Ansible-level backup/restore playbook is still planned. Track progress in `DEPLOYMENT-TODO.md` and update this section once the playbook is available.
## Security
- All external services behind Traefik with HTTPS
- Private registry with BasicAuth
- Secrets managed via Ansible Vault
- Regular security updates via Watchtower
## Troubleshooting
### Check Stack Health
### Infrastructure-Probleme
```bash
cd stacks/<stack-name>
docker compose ps
# Traefik nicht erreichbar
cd deployment/infrastructure/traefik
docker compose logs -f
# PostgreSQL-Verbindungsprobleme
cd deployment/infrastructure/postgresql
docker compose logs postgres
docker network inspect app-internal
```
### Check Service Connectivity
### Application-Probleme
```bash
curl -I https://app.yourdomain.com
# Container-Status prüfen
docker compose -f docker-compose.base.yml -f docker-compose.prod.yml ps
# Logs anzeigen
docker compose -f docker-compose.base.yml -f docker-compose.prod.yml logs -f
# Health Checks
curl https://michaelschiemer.de/health
```
### Network-Probleme
```bash
# Networks prüfen
docker network ls
docker network inspect traefik-public
docker network inspect app-internal
docker network inspect infrastructure
```
### View Logs
```bash
# Application logs (Production)
docker compose -f docker-compose.base.yml -f docker-compose.production.yml logs -f php
## Weitere Dokumentation
# Traefik logs
docker compose -f stacks/traefik/docker-compose.yml logs -f
```
## 📚 Dokumentation Index
**Vollständige Dokumentations-Übersicht:** Siehe [docs/README.md](docs/README.md)
**Wichtigste Dokumente:**
- **[docs/guides/quick-start.md](docs/guides/quick-start.md)** ⭐ - Schnellstart
- **[docs/guides/code-change-workflow.md](docs/guides/code-change-workflow.md)** - Code deployen
- **[docs/reference/application-stack.md](docs/reference/application-stack.md)** - Deployment-Details
- **[docs/status/ci-cd-status.md](docs/status/ci-cd-status.md)** - CI/CD Status
- **[docs/status/deployment-summary.md](docs/status/deployment-summary.md)** - Projekt-Übersicht
- [Infrastructure Layer](infrastructure/README.md) - Infrastruktur-Dokumentation
- [Migration Guide](MIGRATION.md) - Migration vom alten System
- [Secrets Management](infrastructure/SECRETS.md) - Secrets-Verwaltung
## Support
Für spezifische Fragen helfen die folgenden Dokumente weiter:
- [docs/reference/workflow-troubleshooting.md](docs/reference/workflow-troubleshooting.md) – Fehleranalyse für Laufzeiten & Pipelines
- [docs/status/ci-cd-status.md](docs/status/ci-cd-status.md) – Pipeline-Status & Checklisten
- [docs/status/deployment-summary.md](docs/status/deployment-summary.md) – Aktueller Projektüberblick
- [docs/reference/application-stack.md](docs/reference/application-stack.md) – Detaillierte Deployment-Schritte
## License
This deployment configuration is part of the Custom PHP Framework project.
Bei Problemen:
1. Logs sammeln: `docker compose logs > debug_logs.txt`
2. Container-Status: `docker compose ps`
3. Network-Status: `docker network ls`