From c088d08639e2c10630d82b876b7968ccba8fb666 Mon Sep 17 00:00:00 2001 From: Michael Schiemer Date: Fri, 7 Nov 2025 20:17:35 +0100 Subject: [PATCH] test: CI/CD pipeline staging test - Repository Setup automatisiert --- deployment/README.md | 143 +++++++--- deployment/ansible/playbooks/README.md | 65 +++++ .../playbooks/setup-gitea-repository.yml | 254 ++++++++++++++++++ deployment/stacks/gitea/README.md | 55 ++++ 4 files changed, 482 insertions(+), 35 deletions(-) create mode 100644 deployment/ansible/playbooks/README.md create mode 100644 deployment/ansible/playbooks/setup-gitea-repository.yml diff --git a/deployment/README.md b/deployment/README.md index cc886039..73f157cf 100644 --- a/deployment/README.md +++ b/deployment/README.md @@ -92,15 +92,48 @@ deployment/ ## Getting Started +### 🧪 Pipeline-Tests vorbereiten + +**Vor dem ersten Deployment:** + +1. **Prerequisites prüfen:** + ```bash + ./deployment/scripts/test-pipeline-prerequisites.sh + ``` + +2. **Test-Anleitung lesen:** + - [Pipeline Test Checklist](docs/guides/pipeline-test-checklist.md) ⭐ - Schritt-für-Schritt Anleitung + - [Pipeline Testing Guide](docs/guides/pipeline-testing-guide.md) - Übersicht und Troubleshooting + +3. **Backup-Test durchführen:** + ```bash + ./deployment/scripts/test-backup.sh + ``` + ### 🚀 Quick Start: Code deployen -**Einfachste Methode:** +**Empfohlener Workflow (Staging → Production):** + +1. **Push auf `staging` Branch** (Standard für Entwicklung) ```bash git add . git commit -m "feat: Add new feature" -git push origin main # → Automatisches Deployment! +git push origin staging # → Automatisches Deployment zu Staging ``` +2. **Testen auf Staging** +- Staging URL: `https://staging.michaelschiemer.de` +- Tests durchführen und verifizieren + +3. **Merge nach `main`** (nur nach erfolgreichem Test) +```bash +git checkout main +git merge staging +git push origin main # → Automatisches Deployment zu Production +``` + +**⚠️ Wichtig:** Niemals direkt auf `main` pushen - immer erst auf `staging` testen! + **Pipeline-Status:** `https://git.michaelschiemer.de/michael/michaelschiemer/actions` **📖 Vollständige Anleitung:** Siehe [docs/guides/quick-start.md](docs/guides/quick-start.md) oder [docs/guides/code-change-workflow.md](docs/guides/code-change-workflow.md) @@ -134,7 +167,26 @@ Dieses Playbook deployed alle Stacks: - Docker Registry (Private Registry) - Gitea (Git Server) - Monitoring (Portainer, Grafana, Prometheus) -- **Application Stack** (PHP Application + Nginx + Redis + Queue Workers) +- **Production Stack** (PHP Application + Nginx + Redis + Queue Workers) + +**Gitea Initial Setup (nach Infrastructure Deployment):** +```bash +# Automatische Initial Setup via Ansible +cd deployment/ansible + +# 1. Gitea Initial Configuration (Admin-User erstellen) +ansible-playbook -i inventory/production.yml \ + playbooks/setup-gitea-initial-config.yml \ + --vault-password-file secrets/.vault_pass + +# 2. Repository in Gitea erstellen und Git-Remote konfigurieren +ansible-playbook -i inventory/production.yml \ + playbooks/setup-gitea-repository.yml \ + --vault-password-file secrets/.vault_pass \ + -e "repo_name=michaelschiemer" \ + -e "repo_owner=michael" \ + -e "repo_private=false" +``` **📖 Vollständige Setup-Anleitung:** Siehe [SETUP-GUIDE.md](SETUP-GUIDE.md) @@ -145,33 +197,28 @@ Each stack has its own README with detailed configuration: - [Traefik](stacks/traefik/README.md) - Reverse proxy setup - [Gitea](stacks/gitea/README.md) - Git server configuration - [Registry](stacks/registry/README.md) - Private registry setup -- [Application](stacks/application/README.md) - Application deployment +- [Production](stacks/production/README.md) - Production application deployment - [PostgreSQL](stacks/postgresql/README.md) - Database configuration - [Monitoring](stacks/monitoring/README.md) - Monitoring stack ## Deployment Commands -### Code deployen (Image-basiert) +### Automatisches Deployment (Empfohlen) + +**Standard-Workflow: Staging → Production** + +1. **Push auf `staging`** (Standard für Entwicklung) ```bash -cd deployment/ansible -ansible-playbook -i inventory/production.yml \ - playbooks/deploy-update.yml \ - -e "image_tag=abc1234-1696234567" +git add . +git commit -m "feat: Add new feature" +git push origin staging # → Deployt zu Staging ``` -### Code synchen (Git-basiert) +2. **Testen auf Staging**, dann **Merge nach `main`** ```bash -cd deployment/ansible -ansible-playbook -i inventory/production.yml \ - playbooks/sync-code.yml \ - -e "git_branch=main" -``` - -### Rollback zu vorheriger Version -```bash -cd deployment/ansible -ansible-playbook -i inventory/production.yml \ - playbooks/rollback.yml +git checkout main +git merge staging +git push origin main # → Deployt zu Production ``` **📖 Vollständige Command-Referenz:** Siehe [docs/guides/deployment-commands.md](docs/guides/deployment-commands.md) @@ -185,34 +232,54 @@ docker compose up -d ## CI/CD Pipeline -The CI/CD pipeline is defined in `.gitea/workflows/production-deploy.yml` and runs automatically on push to `main` branch. +The CI/CD pipeline is defined in `.gitea/workflows/build-image.yml` and runs automatically on push to `staging` or `main` branch. -### Quick Start: Deploy Code Changes +### Recommended Workflow: Staging → Production +**1. Push to `staging` (Standard for Development)** ```bash -# 1. Make changes locally +# Make changes locally # ... edit files ... -# 2. Commit changes +# Commit and push to staging git add . git commit -m "feat: Add new feature" - -# 3. Push to main → Automatic deployment starts -git push origin main +git push origin staging # → Deploys to Staging ``` -**What happens automatically:** +**What happens automatically on `staging`:** - ✅ Tests run (~2-5 min) - ✅ Docker image is built (~3-5 min) - ✅ Image is pushed to registry (~1-2 min) -- ✅ Ansible deployment runs (~2-4 min) -- ✅ Application stack is updated +- ✅ Deployment to Staging via SSH/SCP (~2-4 min) +- ✅ Staging stack is updated -**Total time:** ~8-15 minutes +**2. Test on Staging** +- Staging URL: `https://staging.michaelschiemer.de` +- Verify functionality and run tests + +**3. Merge to `main` (Only after successful testing)** +```bash +git checkout main +git merge staging +git push origin main # → Deploys to Production +``` + +**What happens automatically on `main`:** +- ✅ Tests run (~2-5 min) +- ✅ Docker image is built (~3-5 min) +- ✅ Image is pushed to registry (~1-2 min) +- ✅ Deployment to Production via SSH/SCP (~2-4 min) +- ✅ Production stack is updated + +**Total time per deployment:** ~8-15 minutes **Status check:** - Pipeline status: `https://git.michaelschiemer.de/michael/michaelschiemer/actions` -- Application status: `ssh deploy@94.16.110.151 "cd ~/deployment/stacks/application && docker compose ps"` +- Staging status: `ssh deploy@94.16.110.151 "cd ~/deployment/stacks/staging && docker compose ps"` +- Production status: `ssh deploy@94.16.110.151 "cd ~/deployment/stacks/production && docker compose ps"` + +**⚠️ Important:** Never push directly to `main` - always test on `staging` first! **📖 Vollständige Dokumentation:** @@ -224,11 +291,17 @@ git push origin main ### Pipeline Details -The CI/CD pipeline runs on push to main branch: +The CI/CD pipeline runs on push to `staging` or `main` branch: +**On `staging` branch:** 1. **Build Stage**: Build Docker image 2. **Push Stage**: Push to private registry -3. **Deploy Stage**: Deploy to production via Ansible +3. **Deploy Stage**: Deploy to Staging via SSH/SCP + +**On `main` branch:** +1. **Build Stage**: Build Docker image +2. **Push Stage**: Push to private registry +3. **Deploy Stage**: Deploy to Production via SSH/SCP ## Monitoring diff --git a/deployment/ansible/playbooks/README.md b/deployment/ansible/playbooks/README.md new file mode 100644 index 00000000..f0c1f8ca --- /dev/null +++ b/deployment/ansible/playbooks/README.md @@ -0,0 +1,65 @@ +# Ansible Playbooks - Übersicht + +## Verfügbare Playbooks + +### Infrastructure Setup +- **`setup-infrastructure.yml`** - Deployed alle Stacks (Traefik, PostgreSQL, Redis, Registry, Gitea, Monitoring, Production) +- **`setup-production-secrets.yml`** - Deployed Secrets zu Production +- **`setup-ssl-certificates.yml`** - SSL Certificate Setup +- **`setup-wireguard-host.yml`** - WireGuard VPN Setup +- **`sync-stacks.yml`** - Synchronisiert Stack-Konfigurationen zum Server + +### Deployment & Updates +- **`rollback.yml`** - Rollback zu vorheriger Version +- **`backup.yml`** - Erstellt Backups von PostgreSQL, Application Data, Gitea, Registry +- **`deploy-image.yml`** - Docker Image Deployment (wird von CI/CD Workflows verwendet) + +### Maintenance +- **`system-maintenance.yml`** - System-Updates, Unattended-Upgrades, Docker-Pruning +- **`troubleshoot.yml`** - Unified Troubleshooting mit Tags +- **`update-gitea-config.yml`** - Aktualisiert Gitea-Konfiguration und startet neu + +### WireGuard +- **`generate-wireguard-client.yml`** - Generiert WireGuard Client-Config +- **`wireguard-routing.yml`** - Konfiguriert WireGuard Routing + +### Initial Deployment +- **`sync-application-code.yml`** - Rsync-basiertes Code-Sync für Initial Deployment (synchronisiert Code vom lokalen Repository zum Server) +- **`deploy-application-code.yml`** - Git-basiertes Code-Deployment (für CI/CD und zukünftige Deployments) +- **`install-composer-dependencies.yml`** - Installiert Composer Dependencies im PHP Container +- **`build-initial-image.yml`** - Build und Push des initialen Docker Images (für erstes Deployment) + +### Code Deployment +- **`sync-application-code.yml`** - Rsync-basiertes Code-Sync (Initial Deployment) +- **`deploy-application-code.yml`** - Git-basiertes Code-Deployment (CI/CD) +- **`deploy-image.yml`** - Docker Image Deployment zu Application Stack + +### Troubleshooting & Diagnostics +- **`check-container-logs.yml`** - Container Logs prüfen (queue-worker, web, scheduler) +- **`check-container-status.yml`** - Container Status prüfen +- **`check-final-status.yml`** - Finale Status-Prüfung aller Container +- **`fix-container-issues.yml`** - Container-Probleme beheben (Composer Dependencies, Permissions) +- **`fix-web-container.yml`** - Web Container Permissions beheben +- **`recreate-containers-with-env.yml`** - Container mit env_file neu erstellen +- **`sync-and-recreate-containers.yml`** - Docker Compose sync und Container recreate + +### CI/CD & Development +- **`setup-gitea-runner-ci.yml`** - Gitea Runner CI Setup +- **`setup-gitea-initial-config.yml`** - Gitea Initial Setup (automatisiert via app.ini + CLI) +- **`setup-gitea-repository.yml`** - Erstellt Repository in Gitea und konfiguriert Git-Remote (automatisiert via API) +- **`install-docker.yml`** - Docker Installation auf Server + +## Entfernte/Legacy Playbooks + +Die folgenden Playbooks wurden entfernt, da sie nicht mehr benötigt werden: +- ~~`build-and-push.yml`~~ - Wird durch CI/CD Pipeline ersetzt +- ~~`remove-framework-production-stack.yml`~~ - Temporäres Playbook +- ~~`remove-temporary-grafana-ip.yml`~~ - Temporäres Playbook + +## Verwendung + +```bash +cd deployment/ansible +ansible-playbook -i inventory/production.yml playbooks/.yml +``` + diff --git a/deployment/ansible/playbooks/setup-gitea-repository.yml b/deployment/ansible/playbooks/setup-gitea-repository.yml new file mode 100644 index 00000000..19221b82 --- /dev/null +++ b/deployment/ansible/playbooks/setup-gitea-repository.yml @@ -0,0 +1,254 @@ +--- +# Ansible Playbook: Setup Gitea Repository +# Purpose: Automatically create repository in Gitea and configure Git remote +# Usage: +# ansible-playbook -i inventory/production.yml playbooks/setup-gitea-repository.yml \ +# --vault-password-file secrets/.vault_pass \ +# -e "repo_name=michaelschiemer" \ +# -e "repo_owner=michael" \ +# -e "repo_private=false" + +- name: Setup Gitea Repository + hosts: production + vars: + ansible_connection: local + gitea_url: "https://{{ gitea_domain | default('git.michaelschiemer.de') }}" + gitea_admin_username: "{{ vault_gitea_admin_username | default('admin') }}" + gitea_admin_password: "{{ vault_gitea_admin_password | default('') }}" + + tasks: + - name: Set repository variables from parameters + set_fact: + repo_name: "{{ repo_name | default('michaelschiemer') }}" + repo_owner: "{{ repo_owner | default('michael') }}" + repo_private: "{{ repo_private | default(false) | bool }}" + repo_description: "{{ repo_description | default('Main application repository') }}" + repo_auto_init: "{{ repo_auto_init | default(false) | bool }}" + configure_git_remote: "{{ configure_git_remote | default(true) | bool }}" + git_repo_path: "{{ git_repo_path | default('/home/michael/dev/michaelschiemer') }}" + + - name: Verify Gitea is accessible + uri: + url: "{{ gitea_url }}" + method: GET + status_code: [200, 302, 502] + validate_certs: false + timeout: 10 + register: gitea_health + failed_when: false + + - name: Debug Gitea health status + debug: + msg: "Gitea health check returned status: {{ gitea_health.status }}" + + - name: Fail if Gitea is not accessible + fail: + msg: "Gitea is not accessible at {{ gitea_url }}. Status: {{ gitea_health.status }}. Please check if Gitea is running." + when: gitea_health.status not in [200, 302, 502] + + - name: Check if API token exists in vault + set_fact: + has_vault_token: "{{ vault_git_token is defined and vault_git_token | length > 0 }}" + no_log: true + + - name: Get or create Gitea API token + uri: + url: "{{ gitea_url }}/api/v1/users/{{ gitea_admin_username }}/tokens" + method: POST + user: "{{ gitea_admin_username }}" + password: "{{ gitea_admin_password }}" + body_format: json + body: + name: "ansible-repo-setup-{{ ansible_date_time.epoch }}" + scopes: + - write:repository + - read:repository + - admin:repo + status_code: [201, 400, 401, 502] + validate_certs: false + force_basic_auth: yes + register: api_token_result + failed_when: false + when: not has_vault_token + no_log: true + + - name: Extract API token from response + set_fact: + gitea_api_token: "{{ api_token_result.json.sha1 | default('') }}" + when: + - not has_vault_token + - api_token_result.status == 201 + - api_token_result.json.sha1 is defined + no_log: true + + - name: Use existing API token from vault + set_fact: + gitea_api_token: "{{ vault_git_token }}" + when: has_vault_token + no_log: true + + - name: Set flag to use basic auth if token creation failed + set_fact: + use_basic_auth: "{{ gitea_api_token | default('') | length == 0 }}" + no_log: true + + - name: Fail if no authentication method available + fail: + msg: "Could not create or retrieve Gitea API token, and admin credentials are not available. Please create a token manually or set vault_git_token in vault." + when: + - use_basic_auth | bool + - gitea_admin_password | default('') | length == 0 + + - name: Initialize repo_check variable + set_fact: + repo_check: {"status": 0} + when: repo_check is not defined + + - name: Check if repository already exists (with token) + uri: + url: "{{ gitea_url }}/api/v1/repos/{{ repo_owner }}/{{ repo_name }}" + method: GET + headers: + Authorization: "token {{ gitea_api_token }}" + status_code: [200, 404, 502] + validate_certs: false + timeout: 10 + register: repo_check_token + when: not use_basic_auth + failed_when: false + + - name: Set repo_check from token result + set_fact: + repo_check: "{{ repo_check_token }}" + when: + - not use_basic_auth + - repo_check_token is defined + + - name: Check if repository already exists (with basic auth) + uri: + url: "{{ gitea_url }}/api/v1/repos/{{ repo_owner }}/{{ repo_name }}" + method: GET + user: "{{ gitea_admin_username }}" + password: "{{ gitea_admin_password }}" + status_code: [200, 404, 502] + validate_certs: false + force_basic_auth: yes + timeout: 10 + register: repo_check_basic + when: use_basic_auth + failed_when: false + no_log: true + + - name: Set repo_check from basic auth result + set_fact: + repo_check: "{{ repo_check_basic }}" + when: + - use_basic_auth + - repo_check_basic is defined + + - name: Debug repo_check status + debug: + msg: "Repository check - Status: {{ repo_check.status | default('undefined') }}, use_basic_auth: {{ use_basic_auth | default('undefined') }}" + + - name: Create repository in Gitea (with token) + uri: + url: "{{ gitea_url }}/api/v1/user/repos" + method: POST + headers: + Authorization: "token {{ gitea_api_token }}" + Content-Type: "application/json" + body_format: json + body: + name: "{{ repo_name }}" + description: "{{ repo_description }}" + private: "{{ repo_private }}" + auto_init: "{{ repo_auto_init }}" + status_code: [201, 409, 502] + validate_certs: false + timeout: 10 + register: repo_create_result + when: + - (repo_check.status | default(0)) in [404, 502, 0] + - not use_basic_auth + failed_when: false + + - name: Create repository in Gitea (with basic auth) + uri: + url: "{{ gitea_url }}/api/v1/user/repos" + method: POST + user: "{{ gitea_admin_username }}" + password: "{{ gitea_admin_password }}" + body_format: json + body: + name: "{{ repo_name }}" + description: "{{ repo_description }}" + private: "{{ repo_private }}" + auto_init: "{{ repo_auto_init }}" + status_code: [201, 409] + validate_certs: false + force_basic_auth: yes + timeout: 10 + register: repo_create_result + when: + - (repo_check.status | default(0)) != 200 + - use_basic_auth + no_log: true + + - name: Display repository creation result + debug: + msg: "Repository {{ repo_owner }}/{{ repo_name }} already exists or was created successfully" + when: repo_check.status | default(0) == 200 or (repo_create_result is defined and repo_create_result.status | default(0) == 201) + + - name: Get repository clone URL + set_fact: + repo_clone_url: "{{ gitea_url | replace('https://', '') | replace('http://', '') }}/{{ repo_owner }}/{{ repo_name }}.git" + repo_https_url: "https://{{ gitea_admin_username }}:{{ gitea_api_token }}@{{ gitea_url | replace('https://', '') | replace('http://', '') }}/{{ repo_owner }}/{{ repo_name }}.git" + + - name: Check if Git repository exists locally + stat: + path: "{{ git_repo_path }}/.git" + register: git_repo_exists + when: configure_git_remote | bool + + - name: Configure Git remote (local) + command: > + git remote set-url origin {{ repo_clone_url }} + args: + chdir: "{{ git_repo_path }}" + register: git_remote_result + when: + - configure_git_remote | bool + - git_repo_path is defined + - git_repo_exists.stat.exists + changed_when: git_remote_result.rc == 0 + failed_when: false + + - name: Add Git remote if it doesn't exist + command: > + git remote add origin {{ repo_clone_url }} + args: + chdir: "{{ git_repo_path }}" + register: git_remote_add_result + when: + - configure_git_remote | bool + - git_repo_path is defined + - git_repo_exists.stat.exists + - git_remote_result.rc != 0 + changed_when: git_remote_add_result.rc == 0 + failed_when: false + + - name: Display success message + debug: + msg: + - "✅ Repository created successfully!" + - "Repository URL: {{ gitea_url }}/{{ repo_owner }}/{{ repo_name }}" + - "Clone URL: {{ repo_clone_url }}" + - "" + - "Next steps:" + - "1. Push your code: git push -u origin staging" + - "2. Monitor pipeline: {{ gitea_url }}/{{ repo_owner }}/{{ repo_name }}/actions" + - "" + - "Note: If you need to push, you may need to authenticate with:" + - " Username: {{ gitea_admin_username }}" + - " Password: (use vault_gitea_admin_password or create a Personal Access Token)" + diff --git a/deployment/stacks/gitea/README.md b/deployment/stacks/gitea/README.md index 06e0fafc..c5b3bc92 100644 --- a/deployment/stacks/gitea/README.md +++ b/deployment/stacks/gitea/README.md @@ -86,6 +86,39 @@ docker compose ps ### First Time Configuration +**Option 1: Automated Setup (Recommended)** + +The Gitea initial setup can be automated using Ansible: + +```bash +cd deployment/ansible + +# 1. Set Gitea admin credentials in vault +ansible-vault edit secrets/production.vault.yml --vault-password-file secrets/.vault_pass + +# Add these variables: +# vault_gitea_admin_username: "admin" +# vault_gitea_admin_password: "your-secure-password" +# vault_gitea_admin_email: "kontakt@michaelschiemer.de" + +# 2. Run the setup playbook +ansible-playbook -i inventory/production.yml \ + playbooks/setup-gitea-initial-config.yml \ + --vault-password-file secrets/.vault_pass +``` + +The playbook will: +- Check if Gitea is already configured +- Generate `app.ini` with `INSTALL_LOCK = true` to skip the initial setup page +- Copy the configuration file to the Gitea container +- Create admin user via Gitea CLI with credentials from vault +- Use database settings from environment variables + +**How it works:** +The playbook creates a complete `app.ini` configuration file with `INSTALL_LOCK = true` in the `[security]` section. This tells Gitea to skip the initial setup wizard. The admin user is then created using the `gitea admin user create` command. + +**Option 2: Manual Setup** + 1. **Access Gitea**: https://git.michaelschiemer.de 2. **Initial Setup Wizard**: @@ -107,6 +140,28 @@ docker compose ps ### Creating a Repository +**Option 1: Automated Setup (Recommended)** + +Use Ansible to automatically create the repository and configure Git remote: + +```bash +cd deployment/ansible + +ansible-playbook -i inventory/production.yml \ + playbooks/setup-gitea-repository.yml \ + --vault-password-file secrets/.vault_pass \ + -e "repo_name=michaelschiemer" \ + -e "repo_owner=michael" \ + -e "repo_private=false" +``` + +The playbook will: +- Create repository in Gitea via API +- Configure Git remote automatically +- Use credentials from Ansible Vault + +**Option 2: Manual Setup** + 1. Log in to https://git.michaelschiemer.de 2. Click "+" → "New Repository" 3. Fill in repository details