test: CI/CD pipeline staging test - Repository Setup automatisiert

deployment/README.md

@@ -92,15 +92,48 @@ deployment/
 
 ## Getting Started
 
+### 🧪 Pipeline-Tests vorbereiten
+
+**Vor dem ersten Deployment:**
+
+1. **Prerequisites prüfen:**
+   ```bash
+   ./deployment/scripts/test-pipeline-prerequisites.sh
+   ```
+
+2. **Test-Anleitung lesen:**
+   - [Pipeline Test Checklist](docs/guides/pipeline-test-checklist.md) ⭐ - Schritt-für-Schritt Anleitung
+   - [Pipeline Testing Guide](docs/guides/pipeline-testing-guide.md) - Übersicht und Troubleshooting
+
+3. **Backup-Test durchführen:**
+   ```bash
+   ./deployment/scripts/test-backup.sh
+   ```
+
 ### 🚀 Quick Start: Code deployen
 
-**Einfachste Methode:**
+**Empfohlener Workflow (Staging → Production):**
+
+1. **Push auf `staging` Branch** (Standard für Entwicklung)
 ```bash
 git add .
 git commit -m "feat: Add new feature"
-git push origin main  # → Automatisches Deployment!
+git push origin staging  # → Automatisches Deployment zu Staging
 ```
 
+2. **Testen auf Staging**
+- Staging URL: `https://staging.michaelschiemer.de`
+- Tests durchführen und verifizieren
+
+3. **Merge nach `main`** (nur nach erfolgreichem Test)
+```bash
+git checkout main
+git merge staging
+git push origin main  # → Automatisches Deployment zu Production
+```
+
+**⚠️ Wichtig:** Niemals direkt auf `main` pushen - immer erst auf `staging` testen!
+
 **Pipeline-Status:** `https://git.michaelschiemer.de/michael/michaelschiemer/actions`
 
 **📖 Vollständige Anleitung:** Siehe [docs/guides/quick-start.md](docs/guides/quick-start.md) oder [docs/guides/code-change-workflow.md](docs/guides/code-change-workflow.md)
@@ -134,7 +167,26 @@ Dieses Playbook deployed alle Stacks:
 - Docker Registry (Private Registry)
 - Gitea (Git Server)
 - Monitoring (Portainer, Grafana, Prometheus)
-- **Application Stack** (PHP Application + Nginx + Redis + Queue Workers)
+- **Production Stack** (PHP Application + Nginx + Redis + Queue Workers)
+
+**Gitea Initial Setup (nach Infrastructure Deployment):**
+```bash
+# Automatische Initial Setup via Ansible
+cd deployment/ansible
+
+# 1. Gitea Initial Configuration (Admin-User erstellen)
+ansible-playbook -i inventory/production.yml \
+  playbooks/setup-gitea-initial-config.yml \
+  --vault-password-file secrets/.vault_pass
+
+# 2. Repository in Gitea erstellen und Git-Remote konfigurieren
+ansible-playbook -i inventory/production.yml \
+  playbooks/setup-gitea-repository.yml \
+  --vault-password-file secrets/.vault_pass \
+  -e "repo_name=michaelschiemer" \
+  -e "repo_owner=michael" \
+  -e "repo_private=false"
+```
 
 **📖 Vollständige Setup-Anleitung:** Siehe [SETUP-GUIDE.md](SETUP-GUIDE.md)
 
@@ -145,33 +197,28 @@ Each stack has its own README with detailed configuration:
 - [Traefik](stacks/traefik/README.md) - Reverse proxy setup
 - [Gitea](stacks/gitea/README.md) - Git server configuration
 - [Registry](stacks/registry/README.md) - Private registry setup
-- [Application](stacks/application/README.md) - Application deployment
+- [Production](stacks/production/README.md) - Production application deployment
 - [PostgreSQL](stacks/postgresql/README.md) - Database configuration
 - [Monitoring](stacks/monitoring/README.md) - Monitoring stack
 
 ## Deployment Commands
 
-### Code deployen (Image-basiert)
+### Automatisches Deployment (Empfohlen)
+
+**Standard-Workflow: Staging → Production**
+
+1. **Push auf `staging`** (Standard für Entwicklung)
 ```bash
-cd deployment/ansible
+git add .
-ansible-playbook -i inventory/production.yml \
+git commit -m "feat: Add new feature"
-  playbooks/deploy-update.yml \
+git push origin staging  # → Deployt zu Staging
-  -e "image_tag=abc1234-1696234567"
 ```
 
-### Code synchen (Git-basiert)
+2. **Testen auf Staging**, dann **Merge nach `main`**
 ```bash
-cd deployment/ansible
+git checkout main
-ansible-playbook -i inventory/production.yml \
+git merge staging
-  playbooks/sync-code.yml \
+git push origin main  # → Deployt zu Production
-  -e "git_branch=main"
-```
-
-### Rollback zu vorheriger Version
-```bash
-cd deployment/ansible
-ansible-playbook -i inventory/production.yml \
-  playbooks/rollback.yml
 ```
 
 **📖 Vollständige Command-Referenz:** Siehe [docs/guides/deployment-commands.md](docs/guides/deployment-commands.md)
@@ -185,34 +232,54 @@ docker compose up -d
 
 ## CI/CD Pipeline
 
-The CI/CD pipeline is defined in `.gitea/workflows/production-deploy.yml` and runs automatically on push to `main` branch.
+The CI/CD pipeline is defined in `.gitea/workflows/build-image.yml` and runs automatically on push to `staging` or `main` branch.
 
-### Quick Start: Deploy Code Changes
+### Recommended Workflow: Staging → Production
 
+**1. Push to `staging` (Standard for Development)**
 ```bash
-# 1. Make changes locally
+# Make changes locally
 # ... edit files ...
 
-# 2. Commit changes
+# Commit and push to staging
 git add .
 git commit -m "feat: Add new feature"
-
+git push origin staging  # → Deploys to Staging
-# 3. Push to main → Automatic deployment starts
-git push origin main
 ```
 
-**What happens automatically:**
+**What happens automatically on `staging`:**
 - ✅ Tests run (~2-5 min)
 - ✅ Docker image is built (~3-5 min)
 - ✅ Image is pushed to registry (~1-2 min)
-- ✅ Ansible deployment runs (~2-4 min)
+- ✅ Deployment to Staging via SSH/SCP (~2-4 min)
-- ✅ Application stack is updated
+- ✅ Staging stack is updated
 
-**Total time:** ~8-15 minutes
+**2. Test on Staging**
+- Staging URL: `https://staging.michaelschiemer.de`
+- Verify functionality and run tests
+
+**3. Merge to `main` (Only after successful testing)**
+```bash
+git checkout main
+git merge staging
+git push origin main  # → Deploys to Production
+```
+
+**What happens automatically on `main`:**
+- ✅ Tests run (~2-5 min)
+- ✅ Docker image is built (~3-5 min)
+- ✅ Image is pushed to registry (~1-2 min)
+- ✅ Deployment to Production via SSH/SCP (~2-4 min)
+- ✅ Production stack is updated
+
+**Total time per deployment:** ~8-15 minutes
 
 **Status check:**
 - Pipeline status: `https://git.michaelschiemer.de/michael/michaelschiemer/actions`
-- Application status: `ssh deploy@94.16.110.151 "cd ~/deployment/stacks/application && docker compose ps"`
+- Staging status: `ssh deploy@94.16.110.151 "cd ~/deployment/stacks/staging && docker compose ps"`
+- Production status: `ssh deploy@94.16.110.151 "cd ~/deployment/stacks/production && docker compose ps"`
+
+**⚠️ Important:** Never push directly to `main` - always test on `staging` first!
 
 **📖 Vollständige Dokumentation:**
 
@@ -224,11 +291,17 @@ git push origin main
 
 ### Pipeline Details
 
-The CI/CD pipeline runs on push to main branch:
+The CI/CD pipeline runs on push to `staging` or `main` branch:
 
+**On `staging` branch:**
 1. **Build Stage**: Build Docker image
 2. **Push Stage**: Push to private registry
-3. **Deploy Stage**: Deploy to production via Ansible
+3. **Deploy Stage**: Deploy to Staging via SSH/SCP
+
+**On `main` branch:**
+1. **Build Stage**: Build Docker image
+2. **Push Stage**: Push to private registry
+3. **Deploy Stage**: Deploy to Production via SSH/SCP
 
 ## Monitoring
 

deployment/ansible/playbooks/README.md

@@ -0,0 +1,65 @@
+# Ansible Playbooks - Übersicht
+
+## Verfügbare Playbooks
+
+### Infrastructure Setup
+- **`setup-infrastructure.yml`** - Deployed alle Stacks (Traefik, PostgreSQL, Redis, Registry, Gitea, Monitoring, Production)
+- **`setup-production-secrets.yml`** - Deployed Secrets zu Production
+- **`setup-ssl-certificates.yml`** - SSL Certificate Setup
+- **`setup-wireguard-host.yml`** - WireGuard VPN Setup
+- **`sync-stacks.yml`** - Synchronisiert Stack-Konfigurationen zum Server
+
+### Deployment & Updates
+- **`rollback.yml`** - Rollback zu vorheriger Version
+- **`backup.yml`** - Erstellt Backups von PostgreSQL, Application Data, Gitea, Registry
+- **`deploy-image.yml`** - Docker Image Deployment (wird von CI/CD Workflows verwendet)
+
+### Maintenance
+- **`system-maintenance.yml`** - System-Updates, Unattended-Upgrades, Docker-Pruning
+- **`troubleshoot.yml`** - Unified Troubleshooting mit Tags
+- **`update-gitea-config.yml`** - Aktualisiert Gitea-Konfiguration und startet neu
+
+### WireGuard
+- **`generate-wireguard-client.yml`** - Generiert WireGuard Client-Config
+- **`wireguard-routing.yml`** - Konfiguriert WireGuard Routing
+
+### Initial Deployment
+- **`sync-application-code.yml`** - Rsync-basiertes Code-Sync für Initial Deployment (synchronisiert Code vom lokalen Repository zum Server)
+- **`deploy-application-code.yml`** - Git-basiertes Code-Deployment (für CI/CD und zukünftige Deployments)
+- **`install-composer-dependencies.yml`** - Installiert Composer Dependencies im PHP Container
+- **`build-initial-image.yml`** - Build und Push des initialen Docker Images (für erstes Deployment)
+
+### Code Deployment
+- **`sync-application-code.yml`** - Rsync-basiertes Code-Sync (Initial Deployment)
+- **`deploy-application-code.yml`** - Git-basiertes Code-Deployment (CI/CD)
+- **`deploy-image.yml`** - Docker Image Deployment zu Application Stack
+
+### Troubleshooting & Diagnostics
+- **`check-container-logs.yml`** - Container Logs prüfen (queue-worker, web, scheduler)
+- **`check-container-status.yml`** - Container Status prüfen
+- **`check-final-status.yml`** - Finale Status-Prüfung aller Container
+- **`fix-container-issues.yml`** - Container-Probleme beheben (Composer Dependencies, Permissions)
+- **`fix-web-container.yml`** - Web Container Permissions beheben
+- **`recreate-containers-with-env.yml`** - Container mit env_file neu erstellen
+- **`sync-and-recreate-containers.yml`** - Docker Compose sync und Container recreate
+
+### CI/CD & Development
+- **`setup-gitea-runner-ci.yml`** - Gitea Runner CI Setup
+- **`setup-gitea-initial-config.yml`** - Gitea Initial Setup (automatisiert via app.ini + CLI)
+- **`setup-gitea-repository.yml`** - Erstellt Repository in Gitea und konfiguriert Git-Remote (automatisiert via API)
+- **`install-docker.yml`** - Docker Installation auf Server
+
+## Entfernte/Legacy Playbooks
+
+Die folgenden Playbooks wurden entfernt, da sie nicht mehr benötigt werden:
+- ~~`build-and-push.yml`~~ - Wird durch CI/CD Pipeline ersetzt
+- ~~`remove-framework-production-stack.yml`~~ - Temporäres Playbook
+- ~~`remove-temporary-grafana-ip.yml`~~ - Temporäres Playbook
+
+## Verwendung
+
+```bash
+cd deployment/ansible
+ansible-playbook -i inventory/production.yml playbooks/<playbook>.yml
+```
+

deployment/ansible/playbooks/setup-gitea-repository.yml

@@ -0,0 +1,254 @@
+---
+# Ansible Playbook: Setup Gitea Repository
+# Purpose: Automatically create repository in Gitea and configure Git remote
+# Usage:
+#   ansible-playbook -i inventory/production.yml playbooks/setup-gitea-repository.yml \
+#     --vault-password-file secrets/.vault_pass \
+#     -e "repo_name=michaelschiemer" \
+#     -e "repo_owner=michael" \
+#     -e "repo_private=false"
+
+- name: Setup Gitea Repository
+  hosts: production
+  vars:
+    ansible_connection: local
+    gitea_url: "https://{{ gitea_domain | default('git.michaelschiemer.de') }}"
+    gitea_admin_username: "{{ vault_gitea_admin_username | default('admin') }}"
+    gitea_admin_password: "{{ vault_gitea_admin_password | default('') }}"
+
+  tasks:
+    - name: Set repository variables from parameters
+      set_fact:
+        repo_name: "{{ repo_name | default('michaelschiemer') }}"
+        repo_owner: "{{ repo_owner | default('michael') }}"
+        repo_private: "{{ repo_private | default(false) | bool }}"
+        repo_description: "{{ repo_description | default('Main application repository') }}"
+        repo_auto_init: "{{ repo_auto_init | default(false) | bool }}"
+        configure_git_remote: "{{ configure_git_remote | default(true) | bool }}"
+        git_repo_path: "{{ git_repo_path | default('/home/michael/dev/michaelschiemer') }}"
+
+    - name: Verify Gitea is accessible
+      uri:
+        url: "{{ gitea_url }}"
+        method: GET
+        status_code: [200, 302, 502]
+        validate_certs: false
+        timeout: 10
+      register: gitea_health
+      failed_when: false
+
+    - name: Debug Gitea health status
+      debug:
+        msg: "Gitea health check returned status: {{ gitea_health.status }}"
+
+    - name: Fail if Gitea is not accessible
+      fail:
+        msg: "Gitea is not accessible at {{ gitea_url }}. Status: {{ gitea_health.status }}. Please check if Gitea is running."
+      when: gitea_health.status not in [200, 302, 502]
+
+    - name: Check if API token exists in vault
+      set_fact:
+        has_vault_token: "{{ vault_git_token is defined and vault_git_token | length > 0 }}"
+      no_log: true
+
+    - name: Get or create Gitea API token
+      uri:
+        url: "{{ gitea_url }}/api/v1/users/{{ gitea_admin_username }}/tokens"
+        method: POST
+        user: "{{ gitea_admin_username }}"
+        password: "{{ gitea_admin_password }}"
+        body_format: json
+        body:
+          name: "ansible-repo-setup-{{ ansible_date_time.epoch }}"
+          scopes:
+            - write:repository
+            - read:repository
+            - admin:repo
+        status_code: [201, 400, 401, 502]
+        validate_certs: false
+        force_basic_auth: yes
+      register: api_token_result
+      failed_when: false
+      when: not has_vault_token
+      no_log: true
+
+    - name: Extract API token from response
+      set_fact:
+        gitea_api_token: "{{ api_token_result.json.sha1 | default('') }}"
+      when: 
+        - not has_vault_token
+        - api_token_result.status == 201
+        - api_token_result.json.sha1 is defined
+      no_log: true
+
+    - name: Use existing API token from vault
+      set_fact:
+        gitea_api_token: "{{ vault_git_token }}"
+      when: has_vault_token
+      no_log: true
+
+    - name: Set flag to use basic auth if token creation failed
+      set_fact:
+        use_basic_auth: "{{ gitea_api_token | default('') | length == 0 }}"
+      no_log: true
+
+    - name: Fail if no authentication method available
+      fail:
+        msg: "Could not create or retrieve Gitea API token, and admin credentials are not available. Please create a token manually or set vault_git_token in vault."
+      when: 
+        - use_basic_auth | bool
+        - gitea_admin_password | default('') | length == 0
+
+    - name: Initialize repo_check variable
+      set_fact:
+        repo_check: {"status": 0}
+      when: repo_check is not defined
+
+    - name: Check if repository already exists (with token)
+      uri:
+        url: "{{ gitea_url }}/api/v1/repos/{{ repo_owner }}/{{ repo_name }}"
+        method: GET
+        headers:
+          Authorization: "token {{ gitea_api_token }}"
+        status_code: [200, 404, 502]
+        validate_certs: false
+        timeout: 10
+      register: repo_check_token
+      when: not use_basic_auth
+      failed_when: false
+
+    - name: Set repo_check from token result
+      set_fact:
+        repo_check: "{{ repo_check_token }}"
+      when: 
+        - not use_basic_auth
+        - repo_check_token is defined
+
+    - name: Check if repository already exists (with basic auth)
+      uri:
+        url: "{{ gitea_url }}/api/v1/repos/{{ repo_owner }}/{{ repo_name }}"
+        method: GET
+        user: "{{ gitea_admin_username }}"
+        password: "{{ gitea_admin_password }}"
+        status_code: [200, 404, 502]
+        validate_certs: false
+        force_basic_auth: yes
+        timeout: 10
+      register: repo_check_basic
+      when: use_basic_auth
+      failed_when: false
+      no_log: true
+
+    - name: Set repo_check from basic auth result
+      set_fact:
+        repo_check: "{{ repo_check_basic }}"
+      when: 
+        - use_basic_auth
+        - repo_check_basic is defined
+
+    - name: Debug repo_check status
+      debug:
+        msg: "Repository check - Status: {{ repo_check.status | default('undefined') }}, use_basic_auth: {{ use_basic_auth | default('undefined') }}"
+
+    - name: Create repository in Gitea (with token)
+      uri:
+        url: "{{ gitea_url }}/api/v1/user/repos"
+        method: POST
+        headers:
+          Authorization: "token {{ gitea_api_token }}"
+          Content-Type: "application/json"
+        body_format: json
+        body:
+          name: "{{ repo_name }}"
+          description: "{{ repo_description }}"
+          private: "{{ repo_private }}"
+          auto_init: "{{ repo_auto_init }}"
+        status_code: [201, 409, 502]
+        validate_certs: false
+        timeout: 10
+      register: repo_create_result
+      when: 
+        - (repo_check.status | default(0)) in [404, 502, 0]
+        - not use_basic_auth
+      failed_when: false
+
+    - name: Create repository in Gitea (with basic auth)
+      uri:
+        url: "{{ gitea_url }}/api/v1/user/repos"
+        method: POST
+        user: "{{ gitea_admin_username }}"
+        password: "{{ gitea_admin_password }}"
+        body_format: json
+        body:
+          name: "{{ repo_name }}"
+          description: "{{ repo_description }}"
+          private: "{{ repo_private }}"
+          auto_init: "{{ repo_auto_init }}"
+        status_code: [201, 409]
+        validate_certs: false
+        force_basic_auth: yes
+        timeout: 10
+      register: repo_create_result
+      when: 
+        - (repo_check.status | default(0)) != 200
+        - use_basic_auth
+      no_log: true
+
+    - name: Display repository creation result
+      debug:
+        msg: "Repository {{ repo_owner }}/{{ repo_name }} already exists or was created successfully"
+      when: repo_check.status | default(0) == 200 or (repo_create_result is defined and repo_create_result.status | default(0) == 201)
+
+    - name: Get repository clone URL
+      set_fact:
+        repo_clone_url: "{{ gitea_url | replace('https://', '') | replace('http://', '') }}/{{ repo_owner }}/{{ repo_name }}.git"
+        repo_https_url: "https://{{ gitea_admin_username }}:{{ gitea_api_token }}@{{ gitea_url | replace('https://', '') | replace('http://', '') }}/{{ repo_owner }}/{{ repo_name }}.git"
+
+    - name: Check if Git repository exists locally
+      stat:
+        path: "{{ git_repo_path }}/.git"
+      register: git_repo_exists
+      when: configure_git_remote | bool
+
+    - name: Configure Git remote (local)
+      command: >
+        git remote set-url origin {{ repo_clone_url }}
+      args:
+        chdir: "{{ git_repo_path }}"
+      register: git_remote_result
+      when: 
+        - configure_git_remote | bool
+        - git_repo_path is defined
+        - git_repo_exists.stat.exists
+      changed_when: git_remote_result.rc == 0
+      failed_when: false
+
+    - name: Add Git remote if it doesn't exist
+      command: >
+        git remote add origin {{ repo_clone_url }}
+      args:
+        chdir: "{{ git_repo_path }}"
+      register: git_remote_add_result
+      when: 
+        - configure_git_remote | bool
+        - git_repo_path is defined
+        - git_repo_exists.stat.exists
+        - git_remote_result.rc != 0
+      changed_when: git_remote_add_result.rc == 0
+      failed_when: false
+
+    - name: Display success message
+      debug:
+        msg:
+          - "✅ Repository created successfully!"
+          - "Repository URL: {{ gitea_url }}/{{ repo_owner }}/{{ repo_name }}"
+          - "Clone URL: {{ repo_clone_url }}"
+          - ""
+          - "Next steps:"
+          - "1. Push your code: git push -u origin staging"
+          - "2. Monitor pipeline: {{ gitea_url }}/{{ repo_owner }}/{{ repo_name }}/actions"
+          - ""
+          - "Note: If you need to push, you may need to authenticate with:"
+          - "   Username: {{ gitea_admin_username }}"
+          - "   Password: (use vault_gitea_admin_password or create a Personal Access Token)"
+

deployment/stacks/gitea/README.md

@@ -86,6 +86,39 @@ docker compose ps
 
 ### First Time Configuration
 
+**Option 1: Automated Setup (Recommended)**
+
+The Gitea initial setup can be automated using Ansible:
+
+```bash
+cd deployment/ansible
+
+# 1. Set Gitea admin credentials in vault
+ansible-vault edit secrets/production.vault.yml --vault-password-file secrets/.vault_pass
+
+# Add these variables:
+# vault_gitea_admin_username: "admin"
+# vault_gitea_admin_password: "your-secure-password"
+# vault_gitea_admin_email: "kontakt@michaelschiemer.de"
+
+# 2. Run the setup playbook
+ansible-playbook -i inventory/production.yml \
+  playbooks/setup-gitea-initial-config.yml \
+  --vault-password-file secrets/.vault_pass
+```
+
+The playbook will:
+- Check if Gitea is already configured
+- Generate `app.ini` with `INSTALL_LOCK = true` to skip the initial setup page
+- Copy the configuration file to the Gitea container
+- Create admin user via Gitea CLI with credentials from vault
+- Use database settings from environment variables
+
+**How it works:**
+The playbook creates a complete `app.ini` configuration file with `INSTALL_LOCK = true` in the `[security]` section. This tells Gitea to skip the initial setup wizard. The admin user is then created using the `gitea admin user create` command.
+
+**Option 2: Manual Setup**
+
 1. **Access Gitea**: https://git.michaelschiemer.de
 
 2. **Initial Setup Wizard**:
@@ -107,6 +140,28 @@ docker compose ps
 
 ### Creating a Repository
 
+**Option 1: Automated Setup (Recommended)**
+
+Use Ansible to automatically create the repository and configure Git remote:
+
+```bash
+cd deployment/ansible
+
+ansible-playbook -i inventory/production.yml \
+  playbooks/setup-gitea-repository.yml \
+  --vault-password-file secrets/.vault_pass \
+  -e "repo_name=michaelschiemer" \
+  -e "repo_owner=michael" \
+  -e "repo_private=false"
+```
+
+The playbook will:
+- Create repository in Gitea via API
+- Configure Git remote automatically
+- Use credentials from Ansible Vault
+
+**Option 2: Manual Setup**
+
 1. Log in to https://git.michaelschiemer.de
 2. Click "+" → "New Repository"
 3. Fill in repository details