michael/michaelschiemer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test: CI/CD pipeline staging test - Repository Setup automatisiert

Browse Source
This commit is contained in:
Michael Schiemer
2025-11-07 20:17:35 +01:00
parent 07c054b5ff
commit c088d08639
4 changed files with 482 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files

143
deployment/README.md
View File

@@ -92,15 +92,48 @@ deployment/
## Getting Started
### 🧪 Pipeline-Tests vorbereiten
**Vor dem ersten Deployment:**
1. **Prerequisites prüfen:**
```bash
./deployment/scripts/test-pipeline-prerequisites.sh
```
2. **Test-Anleitung lesen:**
- [Pipeline Test Checklist](docs/guides/pipeline-test-checklist.md) ⭐ - Schritt-für-Schritt Anleitung
- [Pipeline Testing Guide](docs/guides/pipeline-testing-guide.md) - Übersicht und Troubleshooting
3. **Backup-Test durchführen:**
```bash
./deployment/scripts/test-backup.sh
```
### 🚀 Quick Start: Code deployen
**Einfachste Methode:**
**Empfohlener Workflow (Staging → Production):**
1. **Push auf `staging` Branch** (Standard für Entwicklung)
```bash
git add .
git commit -m "feat: Add new feature"
git push origin main # → Automatisches Deployment!
git push origin staging # → Automatisches Deployment zu Staging
```
2. **Testen auf Staging**
- Staging URL: `https://staging.michaelschiemer.de`
- Tests durchführen und verifizieren
3. **Merge nach `main`** (nur nach erfolgreichem Test)
```bash
git checkout main
git merge staging
git push origin main # → Automatisches Deployment zu Production
```
**⚠️ Wichtig:** Niemals direkt auf `main` pushen - immer erst auf `staging` testen!
**Pipeline-Status:** `https://git.michaelschiemer.de/michael/michaelschiemer/actions`
**📖 Vollständige Anleitung:** Siehe [docs/guides/quick-start.md](docs/guides/quick-start.md) oder [docs/guides/code-change-workflow.md](docs/guides/code-change-workflow.md)
@@ -134,7 +167,26 @@ Dieses Playbook deployed alle Stacks:
- Docker Registry (Private Registry)
- Gitea (Git Server)
- Monitoring (Portainer, Grafana, Prometheus)
- **Application Stack** (PHP Application + Nginx + Redis + Queue Workers)
- **Production Stack** (PHP Application + Nginx + Redis + Queue Workers)
**Gitea Initial Setup (nach Infrastructure Deployment):**
```bash
# Automatische Initial Setup via Ansible
cd deployment/ansible
# 1. Gitea Initial Configuration (Admin-User erstellen)
ansible-playbook -i inventory/production.yml \
playbooks/setup-gitea-initial-config.yml \
--vault-password-file secrets/.vault_pass
# 2. Repository in Gitea erstellen und Git-Remote konfigurieren
ansible-playbook -i inventory/production.yml \
playbooks/setup-gitea-repository.yml \
--vault-password-file secrets/.vault_pass \
-e "repo_name=michaelschiemer" \
-e "repo_owner=michael" \
-e "repo_private=false"
```
**📖 Vollständige Setup-Anleitung:** Siehe [SETUP-GUIDE.md](SETUP-GUIDE.md)
@@ -145,33 +197,28 @@ Each stack has its own README with detailed configuration:
- [Traefik](stacks/traefik/README.md) - Reverse proxy setup
- [Gitea](stacks/gitea/README.md) - Git server configuration
- [Registry](stacks/registry/README.md) - Private registry setup
- [Application](stacks/application/README.md) - Application deployment
- [Production](stacks/production/README.md) - Production application deployment
- [PostgreSQL](stacks/postgresql/README.md) - Database configuration
- [Monitoring](stacks/monitoring/README.md) - Monitoring stack
## Deployment Commands
### Code deployen (Image-basiert)
### Automatisches Deployment (Empfohlen)
**Standard-Workflow: Staging → Production**
1. **Push auf `staging`** (Standard für Entwicklung)
```bash
cd deployment/ansible
ansible-playbook -i inventory/production.yml \
playbooks/deploy-update.yml \
-e "image_tag=abc1234-1696234567"
git add .
git commit -m "feat: Add new feature"
git push origin staging # → Deployt zu Staging
```
### Code synchen (Git-basiert)
2. **Testen auf Staging**, dann **Merge nach `main`**
```bash
cd deployment/ansible
ansible-playbook -i inventory/production.yml \
playbooks/sync-code.yml \
-e "git_branch=main"
```
### Rollback zu vorheriger Version
```bash
cd deployment/ansible
ansible-playbook -i inventory/production.yml \
playbooks/rollback.yml
git checkout main
git merge staging
git push origin main # → Deployt zu Production
```
**📖 Vollständige Command-Referenz:** Siehe [docs/guides/deployment-commands.md](docs/guides/deployment-commands.md)
@@ -185,34 +232,54 @@ docker compose up -d
## CI/CD Pipeline
The CI/CD pipeline is defined in `.gitea/workflows/production-deploy.yml` and runs automatically on push to `main` branch.
The CI/CD pipeline is defined in `.gitea/workflows/build-image.yml` and runs automatically on push to `staging` or `main` branch.
### Quick Start: Deploy Code Changes
### Recommended Workflow: Staging → Production
**1. Push to `staging` (Standard for Development)**
```bash
# 1. Make changes locally
# Make changes locally
# ... edit files ...
# 2. Commit changes
# Commit and push to staging
git add .
git commit -m "feat: Add new feature"
# 3. Push to main → Automatic deployment starts
git push origin main
git push origin staging # → Deploys to Staging
```
**What happens automatically:**
**What happens automatically on `staging`:**
- ✅ Tests run (~2-5 min)
- ✅ Docker image is built (~3-5 min)
- ✅ Image is pushed to registry (~1-2 min)
- ✅ Ansible deployment runs (~2-4 min)
- ✅ Application stack is updated
- ✅ Deployment to Staging via SSH/SCP (~2-4 min)
- ✅ Staging stack is updated
**Total time:** ~8-15 minutes
**2. Test on Staging**
- Staging URL: `https://staging.michaelschiemer.de`
- Verify functionality and run tests
**3. Merge to `main` (Only after successful testing)**
```bash
git checkout main
git merge staging
git push origin main # → Deploys to Production
```
**What happens automatically on `main`:**
- ✅ Tests run (~2-5 min)
- ✅ Docker image is built (~3-5 min)
- ✅ Image is pushed to registry (~1-2 min)
- ✅ Deployment to Production via SSH/SCP (~2-4 min)
- ✅ Production stack is updated
**Total time per deployment:** ~8-15 minutes
**Status check:**
- Pipeline status: `https://git.michaelschiemer.de/michael/michaelschiemer/actions`
- Application status: `ssh deploy@94.16.110.151 "cd ~/deployment/stacks/application && docker compose ps"`
- Staging status: `ssh deploy@94.16.110.151 "cd ~/deployment/stacks/staging && docker compose ps"`
- Production status: `ssh deploy@94.16.110.151 "cd ~/deployment/stacks/production && docker compose ps"`
**⚠️ Important:** Never push directly to `main` - always test on `staging` first!
**📖 Vollständige Dokumentation:**
@@ -224,11 +291,17 @@ git push origin main
### Pipeline Details
The CI/CD pipeline runs on push to main branch:
The CI/CD pipeline runs on push to `staging` or `main` branch:
**On `staging` branch:**
1. **Build Stage**: Build Docker image
2. **Push Stage**: Push to private registry
3. **Deploy Stage**: Deploy to production via Ansible
3. **Deploy Stage**: Deploy to Staging via SSH/SCP
**On `main` branch:**
1. **Build Stage**: Build Docker image
2. **Push Stage**: Push to private registry
3. **Deploy Stage**: Deploy to Production via SSH/SCP
## Monitoring

65
deployment/ansible/playbooks/README.md Normal file
View File

@@ -0,0 +1,65 @@
# Ansible Playbooks - Übersicht
## Verfügbare Playbooks
### Infrastructure Setup
- **`setup-infrastructure.yml`** - Deployed alle Stacks (Traefik, PostgreSQL, Redis, Registry, Gitea, Monitoring, Production)
- **`setup-production-secrets.yml`** - Deployed Secrets zu Production
- **`setup-ssl-certificates.yml`** - SSL Certificate Setup
- **`setup-wireguard-host.yml`** - WireGuard VPN Setup
- **`sync-stacks.yml`** - Synchronisiert Stack-Konfigurationen zum Server
### Deployment & Updates
- **`rollback.yml`** - Rollback zu vorheriger Version
- **`backup.yml`** - Erstellt Backups von PostgreSQL, Application Data, Gitea, Registry
- **`deploy-image.yml`** - Docker Image Deployment (wird von CI/CD Workflows verwendet)
### Maintenance
- **`system-maintenance.yml`** - System-Updates, Unattended-Upgrades, Docker-Pruning
- **`troubleshoot.yml`** - Unified Troubleshooting mit Tags
- **`update-gitea-config.yml`** - Aktualisiert Gitea-Konfiguration und startet neu
### WireGuard
- **`generate-wireguard-client.yml`** - Generiert WireGuard Client-Config
- **`wireguard-routing.yml`** - Konfiguriert WireGuard Routing
### Initial Deployment
- **`sync-application-code.yml`** - Rsync-basiertes Code-Sync für Initial Deployment (synchronisiert Code vom lokalen Repository zum Server)
- **`deploy-application-code.yml`** - Git-basiertes Code-Deployment (für CI/CD und zukünftige Deployments)
- **`install-composer-dependencies.yml`** - Installiert Composer Dependencies im PHP Container
- **`build-initial-image.yml`** - Build und Push des initialen Docker Images (für erstes Deployment)
### Code Deployment
- **`sync-application-code.yml`** - Rsync-basiertes Code-Sync (Initial Deployment)
- **`deploy-application-code.yml`** - Git-basiertes Code-Deployment (CI/CD)
- **`deploy-image.yml`** - Docker Image Deployment zu Application Stack
### Troubleshooting & Diagnostics
- **`check-container-logs.yml`** - Container Logs prüfen (queue-worker, web, scheduler)
- **`check-container-status.yml`** - Container Status prüfen
- **`check-final-status.yml`** - Finale Status-Prüfung aller Container
- **`fix-container-issues.yml`** - Container-Probleme beheben (Composer Dependencies, Permissions)
- **`fix-web-container.yml`** - Web Container Permissions beheben
- **`recreate-containers-with-env.yml`** - Container mit env_file neu erstellen
- **`sync-and-recreate-containers.yml`** - Docker Compose sync und Container recreate
### CI/CD & Development
- **`setup-gitea-runner-ci.yml`** - Gitea Runner CI Setup
- **`setup-gitea-initial-config.yml`** - Gitea Initial Setup (automatisiert via app.ini + CLI)
- **`setup-gitea-repository.yml`** - Erstellt Repository in Gitea und konfiguriert Git-Remote (automatisiert via API)
- **`install-docker.yml`** - Docker Installation auf Server
## Entfernte/Legacy Playbooks
Die folgenden Playbooks wurden entfernt, da sie nicht mehr benötigt werden:
- ~~`build-and-push.yml`~~ - Wird durch CI/CD Pipeline ersetzt
- ~~`remove-framework-production-stack.yml`~~ - Temporäres Playbook
- ~~`remove-temporary-grafana-ip.yml`~~ - Temporäres Playbook
## Verwendung
```bash
cd deployment/ansible
ansible-playbook -i inventory/production.yml playbooks/<playbook>.yml
```

254
deployment/ansible/playbooks/setup-gitea-repository.yml Normal file
View File

@@ -0,0 +1,254 @@
---
# Ansible Playbook: Setup Gitea Repository
# Purpose: Automatically create repository in Gitea and configure Git remote
# Usage:
# ansible-playbook -i inventory/production.yml playbooks/setup-gitea-repository.yml \
# --vault-password-file secrets/.vault_pass \
# -e "repo_name=michaelschiemer" \
# -e "repo_owner=michael" \
# -e "repo_private=false"
- name: Setup Gitea Repository
hosts: production
vars:
ansible_connection: local
gitea_url: "https://{{ gitea_domain | default('git.michaelschiemer.de') }}"
gitea_admin_username: "{{ vault_gitea_admin_username | default('admin') }}"
gitea_admin_password: "{{ vault_gitea_admin_password | default('') }}"
tasks:
- name: Set repository variables from parameters
set_fact:
repo_name: "{{ repo_name | default('michaelschiemer') }}"
repo_owner: "{{ repo_owner | default('michael') }}"
repo_private: "{{ repo_private | default(false) | bool }}"
repo_description: "{{ repo_description | default('Main application repository') }}"
repo_auto_init: "{{ repo_auto_init | default(false) | bool }}"
configure_git_remote: "{{ configure_git_remote | default(true) | bool }}"
git_repo_path: "{{ git_repo_path | default('/home/michael/dev/michaelschiemer') }}"
- name: Verify Gitea is accessible
uri:
url: "{{ gitea_url }}"
method: GET
status_code: [200, 302, 502]
validate_certs: false
timeout: 10
register: gitea_health
failed_when: false
- name: Debug Gitea health status
debug:
msg: "Gitea health check returned status: {{ gitea_health.status }}"
- name: Fail if Gitea is not accessible
fail:
msg: "Gitea is not accessible at {{ gitea_url }}. Status: {{ gitea_health.status }}. Please check if Gitea is running."
when: gitea_health.status not in [200, 302, 502]
- name: Check if API token exists in vault
set_fact:
has_vault_token: "{{ vault_git_token is defined and vault_git_token | length > 0 }}"
no_log: true
- name: Get or create Gitea API token
uri:
url: "{{ gitea_url }}/api/v1/users/{{ gitea_admin_username }}/tokens"
method: POST
user: "{{ gitea_admin_username }}"
password: "{{ gitea_admin_password }}"
body_format: json
body:
name: "ansible-repo-setup-{{ ansible_date_time.epoch }}"
scopes:
- write:repository
- read:repository
- admin:repo
status_code: [201, 400, 401, 502]
validate_certs: false
force_basic_auth: yes
register: api_token_result
failed_when: false
when: not has_vault_token
no_log: true
- name: Extract API token from response
set_fact:
gitea_api_token: "{{ api_token_result.json.sha1 | default('') }}"
when:
- not has_vault_token
- api_token_result.status == 201
- api_token_result.json.sha1 is defined
no_log: true
- name: Use existing API token from vault
set_fact:
gitea_api_token: "{{ vault_git_token }}"
when: has_vault_token
no_log: true
- name: Set flag to use basic auth if token creation failed
set_fact:
use_basic_auth: "{{ gitea_api_token | default('') | length == 0 }}"
no_log: true
- name: Fail if no authentication method available
fail:
msg: "Could not create or retrieve Gitea API token, and admin credentials are not available. Please create a token manually or set vault_git_token in vault."
when:
- use_basic_auth | bool
- gitea_admin_password | default('') | length == 0
- name: Initialize repo_check variable
set_fact:
repo_check: {"status": 0}
when: repo_check is not defined
- name: Check if repository already exists (with token)
uri:
url: "{{ gitea_url }}/api/v1/repos/{{ repo_owner }}/{{ repo_name }}"
method: GET
headers:
Authorization: "token {{ gitea_api_token }}"
status_code: [200, 404, 502]
validate_certs: false
timeout: 10
register: repo_check_token
when: not use_basic_auth
failed_when: false
- name: Set repo_check from token result
set_fact:
repo_check: "{{ repo_check_token }}"
when:
- not use_basic_auth
- repo_check_token is defined
- name: Check if repository already exists (with basic auth)
uri:
url: "{{ gitea_url }}/api/v1/repos/{{ repo_owner }}/{{ repo_name }}"
method: GET
user: "{{ gitea_admin_username }}"
password: "{{ gitea_admin_password }}"
status_code: [200, 404, 502]
validate_certs: false
force_basic_auth: yes
timeout: 10
register: repo_check_basic
when: use_basic_auth
failed_when: false
no_log: true
- name: Set repo_check from basic auth result
set_fact:
repo_check: "{{ repo_check_basic }}"
when:
- use_basic_auth
- repo_check_basic is defined
- name: Debug repo_check status
debug:
msg: "Repository check - Status: {{ repo_check.status | default('undefined') }}, use_basic_auth: {{ use_basic_auth | default('undefined') }}"
- name: Create repository in Gitea (with token)
uri:
url: "{{ gitea_url }}/api/v1/user/repos"
method: POST
headers:
Authorization: "token {{ gitea_api_token }}"
Content-Type: "application/json"
body_format: json
body:
name: "{{ repo_name }}"
description: "{{ repo_description }}"
private: "{{ repo_private }}"
auto_init: "{{ repo_auto_init }}"
status_code: [201, 409, 502]
validate_certs: false
timeout: 10
register: repo_create_result
when:
- (repo_check.status | default(0)) in [404, 502, 0]
- not use_basic_auth
failed_when: false
- name: Create repository in Gitea (with basic auth)
uri:
url: "{{ gitea_url }}/api/v1/user/repos"
method: POST
user: "{{ gitea_admin_username }}"
password: "{{ gitea_admin_password }}"
body_format: json
body:
name: "{{ repo_name }}"
description: "{{ repo_description }}"
private: "{{ repo_private }}"
auto_init: "{{ repo_auto_init }}"
status_code: [201, 409]
validate_certs: false
force_basic_auth: yes
timeout: 10
register: repo_create_result
when:
- (repo_check.status | default(0)) != 200
- use_basic_auth
no_log: true
- name: Display repository creation result
debug:
msg: "Repository {{ repo_owner }}/{{ repo_name }} already exists or was created successfully"
when: repo_check.status | default(0) == 200 or (repo_create_result is defined and repo_create_result.status | default(0) == 201)
- name: Get repository clone URL
set_fact:
repo_clone_url: "{{ gitea_url | replace('https://', '') | replace('http://', '') }}/{{ repo_owner }}/{{ repo_name }}.git"
repo_https_url: "https://{{ gitea_admin_username }}:{{ gitea_api_token }}@{{ gitea_url | replace('https://', '') | replace('http://', '') }}/{{ repo_owner }}/{{ repo_name }}.git"
- name: Check if Git repository exists locally
stat:
path: "{{ git_repo_path }}/.git"
register: git_repo_exists
when: configure_git_remote | bool
- name: Configure Git remote (local)
command: >
git remote set-url origin {{ repo_clone_url }}
args:
chdir: "{{ git_repo_path }}"
register: git_remote_result
when:
- configure_git_remote | bool
- git_repo_path is defined
- git_repo_exists.stat.exists
changed_when: git_remote_result.rc == 0
failed_when: false
- name: Add Git remote if it doesn't exist
command: >
git remote add origin {{ repo_clone_url }}
args:
chdir: "{{ git_repo_path }}"
register: git_remote_add_result
when:
- configure_git_remote | bool
- git_repo_path is defined
- git_repo_exists.stat.exists
- git_remote_result.rc != 0
changed_when: git_remote_add_result.rc == 0
failed_when: false
- name: Display success message
debug:
msg:
- "✅ Repository created successfully!"
- "Repository URL: {{ gitea_url }}/{{ repo_owner }}/{{ repo_name }}"
- "Clone URL: {{ repo_clone_url }}"
- ""
- "Next steps:"
- "1. Push your code: git push -u origin staging"
- "2. Monitor pipeline: {{ gitea_url }}/{{ repo_owner }}/{{ repo_name }}/actions"
- ""
- "Note: If you need to push, you may need to authenticate with:"
- " Username: {{ gitea_admin_username }}"
- " Password: (use vault_gitea_admin_password or create a Personal Access Token)"

55
deployment/stacks/gitea/README.md
View File

@@ -86,6 +86,39 @@ docker compose ps
### First Time Configuration
**Option 1: Automated Setup (Recommended)**
The Gitea initial setup can be automated using Ansible:
```bash
cd deployment/ansible
# 1. Set Gitea admin credentials in vault
ansible-vault edit secrets/production.vault.yml --vault-password-file secrets/.vault_pass
# Add these variables:
# vault_gitea_admin_username: "admin"
# vault_gitea_admin_password: "your-secure-password"
# vault_gitea_admin_email: "kontakt@michaelschiemer.de"
# 2. Run the setup playbook
ansible-playbook -i inventory/production.yml \
playbooks/setup-gitea-initial-config.yml \
--vault-password-file secrets/.vault_pass
```
The playbook will:
- Check if Gitea is already configured
- Generate `app.ini` with `INSTALL_LOCK = true` to skip the initial setup page
- Copy the configuration file to the Gitea container
- Create admin user via Gitea CLI with credentials from vault
- Use database settings from environment variables
**How it works:**
The playbook creates a complete `app.ini` configuration file with `INSTALL_LOCK = true` in the `[security]` section. This tells Gitea to skip the initial setup wizard. The admin user is then created using the `gitea admin user create` command.
**Option 2: Manual Setup**
1. **Access Gitea**: https://git.michaelschiemer.de
2. **Initial Setup Wizard**:
@@ -107,6 +140,28 @@ docker compose ps
### Creating a Repository
**Option 1: Automated Setup (Recommended)**
Use Ansible to automatically create the repository and configure Git remote:
```bash
cd deployment/ansible
ansible-playbook -i inventory/production.yml \
playbooks/setup-gitea-repository.yml \
--vault-password-file secrets/.vault_pass \
-e "repo_name=michaelschiemer" \
-e "repo_owner=michael" \
-e "repo_private=false"
```
The playbook will:
- Create repository in Gitea via API
- Configure Git remote automatically
- Use credentials from Ansible Vault
**Option 2: Manual Setup**
1. Log in to https://git.michaelschiemer.de
2. Click "+" → "New Repository"
3. Fill in repository details