feat: CI/CD pipeline setup complete - Ansible playbooks updated, secrets configured, workflow ready

This commit is contained in:
2025-10-31 01:39:24 +01:00
parent 55c04e4fd0
commit e26eb2aa12
601 changed files with 44184 additions and 32477 deletions

View File

@@ -0,0 +1,22 @@
# Gitea Actions Runner Configuration
# Gitea Instance URL (must be accessible from runner)
GITEA_INSTANCE_URL=https://git.michaelschiemer.de
# Runner Registration Token (get from Gitea: Admin > Actions > Runners)
# To generate: Gitea UI > Site Administration > Actions > Runners > Create New Runner
GITEA_RUNNER_REGISTRATION_TOKEN=
# Runner Name (appears in Gitea UI)
GITEA_RUNNER_NAME=dev-runner-01
# Runner Labels (comma-separated)
# Format: label:image
# Example: ubuntu-latest:docker://node:16-bullseye
GITEA_RUNNER_LABELS=ubuntu-latest:docker://node:16-bullseye,ubuntu-22.04:docker://node:16-bullseye,debian-latest:docker://debian:bullseye
# Optional: Custom Docker registry for job images
# DOCKER_REGISTRY_MIRROR=https://registry.michaelschiemer.de
# Optional: Runner capacity (max concurrent jobs)
# GITEA_RUNNER_CAPACITY=1

8
deployment/gitea-runner/.gitignore vendored Normal file
View File

@@ -0,0 +1,8 @@
# Environment configuration
.env
# Runner data
data/
# Logs
*.log

View File

@@ -0,0 +1,693 @@
# Gitea Actions Runner (Development Machine)
Self-hosted Gitea Actions runner for executing CI/CD workflows on the development machine.
## Overview
This setup provides a Gitea Actions runner that executes CI/CD workflows triggered by repository events in Gitea. The runner runs in Docker and uses Docker-in-Docker (DinD) for isolated job execution.
**Key Features**:
- **Self-Hosted**: Runs on development machine with full control
- **Docker-Based**: Isolated execution environment for jobs
- **Docker-in-Docker**: Jobs run in separate containers for security
- **Multiple Labels**: Support for different workflow environments
- **Auto-Restart**: Automatically restarts on failure
- **Secure**: Isolated network and resource limits
## Prerequisites
- Docker and Docker Compose installed
- Gitea instance running (Stack 2: Gitea)
- Admin access to Gitea for runner registration
- Network connectivity to Gitea instance
## Directory Structure
```
gitea-runner/
├── docker-compose.yml # Service definitions
├── .env.example # Environment template
├── .env # Environment configuration (create from .env.example)
├── config.yaml # Runner configuration
├── register.sh # Registration script
├── unregister.sh # Unregistration script
├── data/ # Runner data (auto-created)
│ └── .runner # Registration info (auto-generated)
└── README.md # This file
```
## Quick Start
### 1. Create Environment File
```bash
cd deployment/gitea-runner
cp .env.example .env
```
### 2. Get Registration Token
1. Go to Gitea admin panel: https://git.michaelschiemer.de/admin
2. Navigate to: **Site Administration > Actions > Runners**
3. Click **"Create New Runner"**
4. Copy the registration token
5. Add token to `.env` file:
```bash
nano .env
# Set GITEA_RUNNER_REGISTRATION_TOKEN=<your-token>
```
### 3. Configure Environment Variables
Edit `.env` and configure:
```bash
# Gitea Instance URL
GITEA_INSTANCE_URL=https://git.michaelschiemer.de
# Registration Token (from step 2)
GITEA_RUNNER_REGISTRATION_TOKEN=<your-token>
# Runner Name (appears in Gitea UI)
GITEA_RUNNER_NAME=dev-runner-01
# Runner Labels (what environments this runner supports)
GITEA_RUNNER_LABELS=ubuntu-latest:docker://node:16-bullseye,ubuntu-22.04:docker://node:16-bullseye
```
### 4. Register Runner
Run the registration script:
```bash
./register.sh
```
This will:
- Start the runner services
- Register the runner with Gitea
- Display runner status
### 5. Verify Registration
Check runner status in Gitea:
- Go to: https://git.michaelschiemer.de/admin/actions/runners
- You should see your runner listed as "Idle" or "Active"
## Configuration
### Runner Labels
Labels define what workflow environments the runner supports. Format: `label:image`
**Common Labels**:
```bash
# Ubuntu with Node.js 16
ubuntu-latest:docker://node:16-bullseye
# Ubuntu 22.04
ubuntu-22.04:docker://node:16-bullseye
# Debian
debian-latest:docker://debian:bullseye
# Custom images from private registry
ubuntu-php:docker://registry.michaelschiemer.de/php:8.3-cli
```
**Example Workflow Using Labels**:
```yaml
# .gitea/workflows/test.yml
name: Test
on: [push]
jobs:
test:
runs-on: ubuntu-latest # Uses runner with this label
steps:
- uses: actions/checkout@v3
- run: npm install
- run: npm test
```
### Runner Capacity
Control how many jobs can run concurrently:
**In `.env`**:
```bash
GITEA_RUNNER_CAPACITY=1 # Max concurrent jobs
```
**In `config.yaml`**:
```yaml
runner:
capacity: 1 # Max concurrent jobs
timeout: 3h # Job timeout
```
### Resource Limits
Configure resource limits in `config.yaml`:
```yaml
container:
resources:
memory_limit: 2147483648 # 2GB
cpu_quota: 100000 # 1 CPU
```
## Usage
### Start Runner
```bash
# Start services
docker compose up -d
# View logs
docker compose logs -f gitea-runner
```
### Stop Runner
```bash
docker compose down
```
### Restart Runner
```bash
docker compose restart gitea-runner
```
### View Logs
```bash
# Follow logs
docker compose logs -f gitea-runner
# View last 100 lines
docker compose logs --tail=100 gitea-runner
# View Docker-in-Docker logs
docker compose logs -f docker-dind
```
### Check Runner Status
```bash
# Check container status
docker compose ps
# View runner info
docker compose exec gitea-runner cat /data/.runner
```
### Unregister Runner
```bash
./unregister.sh
```
This will:
- Stop the runner services
- Remove registration file
- Optionally remove runner data
**Note**: You may need to manually delete the runner from Gitea UI:
- Go to: https://git.michaelschiemer.de/admin/actions/runners
- Find the runner and click "Delete"
## Workflow Examples
### Basic Node.js Test Workflow
Create `.gitea/workflows/test.yml` in your repository:
```yaml
name: Test
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Install dependencies
run: npm install
- name: Run tests
run: npm test
```
### PHP Test Workflow
```yaml
name: PHP Tests
on: [push]
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Install PHP dependencies
run: |
apt-get update
apt-get install -y php8.3-cli php8.3-mbstring
php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
php composer-setup.php --install-dir=/usr/local/bin --filename=composer
composer install
- name: Run tests
run: ./vendor/bin/pest
```
### Build and Deploy Workflow
```yaml
name: Deploy
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Build Docker image
run: |
docker build -t registry.michaelschiemer.de/app:latest .
- name: Push to registry
run: |
echo "${{ secrets.REGISTRY_PASSWORD }}" | docker login registry.michaelschiemer.de -u admin --password-stdin
docker push registry.michaelschiemer.de/app:latest
- name: Deploy via SSH
run: |
# Add SSH deployment commands here
```
## Troubleshooting
### Runner Not Appearing in Gitea
**Check registration**:
```bash
# Verify registration file exists
ls -la data/.runner
# Check runner logs
docker compose logs gitea-runner
```
**Re-register**:
```bash
./unregister.sh
./register.sh
```
### Jobs Not Starting
**Check runner status**:
```bash
# View logs
docker compose logs -f gitea-runner
# Check if runner is idle
# In Gitea: Admin > Actions > Runners
```
**Common Issues**:
- Runner is offline: Restart runner (`docker compose restart gitea-runner`)
- No matching labels: Verify workflow `runs-on` matches runner labels
- Capacity reached: Increase `GITEA_RUNNER_CAPACITY` or wait for jobs to finish
### Docker-in-Docker Issues
**Check DinD container**:
```bash
# View DinD logs
docker compose logs docker-dind
# Check DinD is running
docker compose ps docker-dind
```
**Restart DinD**:
```bash
docker compose restart docker-dind
```
### Job Timeout
Jobs timing out after 3 hours? Increase timeout in `config.yaml`:
```yaml
runner:
timeout: 6h # Increase to 6 hours
```
### Network Issues
**Cannot reach Gitea**:
```bash
# Test connectivity from runner
docker compose exec gitea-runner wget -O- https://git.michaelschiemer.de
# Check DNS resolution
docker compose exec gitea-runner nslookup git.michaelschiemer.de
```
### Disk Space Issues
**Clean up old job data**:
```bash
# Remove old workspace data
docker compose exec gitea-runner rm -rf /tmp/gitea-runner/*
# Clean up Docker images
docker image prune -a -f
```
## Security Considerations
### 1. Runner Security
- Runner runs with access to Docker socket (required for jobs)
- Jobs execute in isolated containers via Docker-in-Docker
- Network is isolated from other Docker networks
- Resource limits prevent resource exhaustion
### 2. Registration Token
- Registration token has admin privileges
- Store token securely (in `.env` file, not in git)
- Token is only used during registration
- After registration, runner uses generated credentials
### 3. Job Isolation
- Each job runs in a separate container
- Containers are destroyed after job completion
- Docker-in-Docker provides additional isolation layer
- Valid volume mounts are restricted in `config.yaml`
### 4. Secrets Management
**In Gitea**:
- Store secrets in repository settings: Settings > Secrets
- Access in workflows via `${{ secrets.SECRET_NAME }}`
- Secrets are masked in logs
**Example**:
```yaml
steps:
- name: Deploy
run: |
echo "${{ secrets.DEPLOY_KEY }}" > deploy_key
chmod 600 deploy_key
ssh -i deploy_key user@server "deploy.sh"
```
### 5. Network Security
- Runner network is isolated
- Only runner and DinD containers share network
- No external access to runner management
## Maintenance
### Daily Tasks
- Monitor runner logs for errors
- Check disk space usage
- Verify runner appears as "Idle" in Gitea when not running jobs
### Weekly Tasks
- Review completed jobs in Gitea
- Clean up old Docker images: `docker image prune -a`
- Check runner resource usage
### Monthly Tasks
- Update runner image: `docker compose pull && docker compose up -d`
- Review and update runner labels
- Audit workflow performance
### Update Runner
```bash
# Pull latest image
docker compose pull
# Restart with new image
docker compose up -d
# Verify update
docker compose logs -f gitea-runner
```
## Performance Optimization
### Reduce Job Startup Time
**Cache dependencies** in workflows:
```yaml
- name: Cache dependencies
uses: actions/cache@v3
with:
path: ~/.npm
key: ${{ runner.os }}-npm-${{ hashFiles('**/package-lock.json') }}
```
### Optimize Docker Builds
**Use Docker layer caching**:
```yaml
- name: Build with cache
run: |
docker build \
--cache-from registry.michaelschiemer.de/app:cache \
--tag registry.michaelschiemer.de/app:latest \
.
```
### Increase Runner Capacity
For more concurrent jobs:
```bash
# In .env
GITEA_RUNNER_CAPACITY=2 # Allow 2 concurrent jobs
```
**Note**: Ensure development machine has sufficient resources (CPU, RAM, disk).
## Integration with Deployment Stacks
### Stack 2: Gitea Integration
- Runner connects to Gitea for job fetching
- Uses Gitea API for workflow definitions
- Reports job status back to Gitea
### Stack 3: Docker Registry Integration
- Push built images to private registry
- Pull base images from registry for jobs
- Use registry for caching layers
### Stack 4: Application Deployment
- Build and test application code
- Deploy to application stack via SSH
- Trigger stack updates via Ansible
### Stack 5: Database Migrations
- Run database migrations in workflows
- Test database changes before deployment
- Backup database before migrations
### Stack 6: Monitoring Integration
- Monitor runner resource usage via cAdvisor
- Track job execution metrics in Prometheus
- Alert on runner failures via Grafana
## Advanced Configuration
### Custom Docker Registry for Jobs
Use private registry for job images:
```bash
# In .env
DOCKER_REGISTRY_MIRROR=https://registry.michaelschiemer.de
```
**In workflows**:
```yaml
jobs:
test:
runs-on: ubuntu-latest
container:
image: registry.michaelschiemer.de/php:8.3-cli
credentials:
username: admin
password: ${{ secrets.REGISTRY_PASSWORD }}
```
### Multiple Runners
Run multiple runners for different purposes:
```bash
# Production runner
cd deployment/gitea-runner-prod
cp ../gitea-runner/.env.example .env
# Set GITEA_RUNNER_NAME=prod-runner
# Set different labels
./register.sh
# Staging runner
cd deployment/gitea-runner-staging
cp ../gitea-runner/.env.example .env
# Set GITEA_RUNNER_NAME=staging-runner
./register.sh
```
### Custom Job Container Options
In `config.yaml`:
```yaml
container:
# Custom Docker options
options: "--dns 8.8.8.8 --add-host git.michaelschiemer.de:94.16.110.151"
# Custom network mode
network: host
# Enable privileged mode (use cautiously)
privileged: false
```
## Monitoring and Logging
### View Runner Metrics
```bash
# Container resource usage
docker stats gitea-runner
# Detailed metrics
docker compose exec gitea-runner cat /data/metrics
```
### Centralized Logging
Send logs to monitoring stack:
```yaml
# In docker-compose.yml
services:
gitea-runner:
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "3"
```
### Health Checks
```bash
# Check runner health
docker compose exec gitea-runner ps aux | grep act_runner
# Check job queue
# In Gitea: Actions > Jobs
```
## Backup and Recovery
### Backup Runner Configuration
```bash
# Backup registration and config
tar czf gitea-runner-backup-$(date +%Y%m%d).tar.gz \
.env config.yaml data/.runner
```
### Restore Runner
```bash
# Extract backup
tar xzf gitea-runner-backup-YYYYMMDD.tar.gz
# Restart runner
docker compose up -d
```
**Note**: If registration token changed, re-register:
```bash
./unregister.sh
./register.sh
```
## Support
### Documentation
- Gitea Actions: https://docs.gitea.io/en-us/actions/overview/
- Act Runner: https://gitea.com/gitea/act_runner
- GitHub Actions (compatible): https://docs.github.com/en/actions
### Logs
```bash
# Runner logs
docker compose logs -f gitea-runner
# All logs
docker compose logs -f
# Export logs
docker compose logs > runner-logs-$(date +%Y%m%d).log
```
### Health Check
```bash
# Check all components
docker compose ps
# Test runner connection to Gitea
docker compose exec gitea-runner wget -O- https://git.michaelschiemer.de/api/v1/version
```
---
**Setup Status**: ✅ Ready for registration
**Next Steps**:
1. Copy `.env.example` to `.env`
2. Get registration token from Gitea
3. Run `./register.sh`
4. Verify runner appears in Gitea UI
5. Create a test workflow to verify functionality

View File

@@ -0,0 +1,76 @@
# Gitea Actions Runner Configuration
# https://docs.gitea.io/en-us/actions/act-runner/
log:
level: info
runner:
# File to store runner registration information
file: /data/.runner
# Maximum number of concurrent jobs
capacity: 1
# Timeout for a single job
timeout: 3h
# Whether to enable debug mode (skip SSL verification for setup)
insecure: true
# Timeout for fetching job from Gitea
fetch_timeout: 5s
# Interval for fetching jobs
fetch_interval: 2s
cache:
# Enable cache server
enabled: true
# Cache server directory
dir: /data/cache
# Host address for cache server
host: ""
# Port for cache server
port: 0
container:
# Docker network mode for job containers
network: bridge
# Privileged mode for job containers
privileged: false
# Container options
options: ""
# Working directory in container
workdir_parent: /workspace
# Force pull images before running jobs
force_pull: false
# Default image if not specified in workflow
default_image: node:16-bullseye
# Docker host
docker_host: ""
# Valid volume paths that can be mounted
valid_volumes:
- /workspace
- /data
# Resources limits
resources:
memory_limit: 0
memory_swap_limit: 0
cpu_quota: 0
cpu_period: 0
cpu_set: ""
host:
# Working directory on host
workdir_parent: /tmp/gitea-runner

View File

@@ -0,0 +1,46 @@
version: '3.8'
services:
gitea-runner:
image: gitea/act_runner:latest
container_name: gitea-runner
restart: unless-stopped
volumes:
- ./data:/data
- /var/run/docker.sock:/var/run/docker.sock
- ./config.yaml:/config.yaml:ro
environment:
- GITEA_INSTANCE_URL=${GITEA_INSTANCE_URL}
- GITEA_RUNNER_REGISTRATION_TOKEN=${GITEA_RUNNER_REGISTRATION_TOKEN}
- GITEA_RUNNER_NAME=${GITEA_RUNNER_NAME:-dev-runner}
- GITEA_RUNNER_LABELS=${GITEA_RUNNER_LABELS:-ubuntu-latest:docker://node:16-bullseye,ubuntu-22.04:docker://node:16-bullseye}
networks:
- gitea-runner
depends_on:
- docker-dind
# Docker-in-Docker for isolated job execution
docker-dind:
image: docker:dind
container_name: gitea-runner-dind
restart: unless-stopped
privileged: true
environment:
- DOCKER_TLS_CERTDIR=/certs
volumes:
- docker-certs:/certs
- docker-data:/var/lib/docker
networks:
- gitea-runner
command: ["dockerd", "--host=unix:///var/run/docker.sock", "--host=tcp://0.0.0.0:2376", "--tlsverify"]
networks:
gitea-runner:
name: gitea-runner
driver: bridge
volumes:
docker-certs:
name: gitea-runner-certs
docker-data:
name: gitea-runner-docker-data

View File

@@ -0,0 +1,117 @@
#!/usr/bin/env bash
set -euo pipefail
# Gitea Actions Runner Registration Script
# This script registers the runner with the Gitea instance
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "$SCRIPT_DIR"
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
echo -e "${GREEN}Gitea Actions Runner Registration${NC}"
echo "======================================"
# Check if .env exists
if [ ! -f .env ]; then
echo -e "${RED}Error: .env file not found${NC}"
echo "Please copy .env.example to .env and configure it"
echo ""
echo " cp .env.example .env"
echo " nano .env"
echo ""
exit 1
fi
# Load environment variables
set -a
source .env
set +a
# Validate required variables
if [ -z "${GITEA_INSTANCE_URL:-}" ]; then
echo -e "${RED}Error: GITEA_INSTANCE_URL is not set in .env${NC}"
exit 1
fi
if [ -z "${GITEA_RUNNER_REGISTRATION_TOKEN:-}" ]; then
echo -e "${RED}Error: GITEA_RUNNER_REGISTRATION_TOKEN is not set in .env${NC}"
echo ""
echo "To get registration token:"
echo "1. Go to Gitea: ${GITEA_INSTANCE_URL}/admin/actions/runners"
echo "2. Click 'Create New Runner'"
echo "3. Copy the registration token"
echo "4. Add it to .env file"
echo ""
exit 1
fi
echo -e "${YELLOW}Configuration:${NC}"
echo " Gitea URL: ${GITEA_INSTANCE_URL}"
echo " Runner Name: ${GITEA_RUNNER_NAME:-dev-runner}"
echo " Labels: ${GITEA_RUNNER_LABELS:-ubuntu-latest:docker://node:16-bullseye}"
echo ""
# Check if runner is already registered
if [ -f ./data/.runner ]; then
echo -e "${YELLOW}Warning: Runner appears to be already registered${NC}"
echo "Registration file exists at: ./data/.runner"
echo ""
read -p "Do you want to re-register? This will unregister the existing runner. (y/N): " -n 1 -r
echo ""
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
echo "Registration cancelled"
exit 0
fi
echo "Removing existing registration..."
rm -f ./data/.runner
fi
# Create data directory if it doesn't exist
mkdir -p ./data
# Start services
echo ""
echo -e "${YELLOW}Starting services...${NC}"
docker compose up -d
# Wait for services to be ready
echo "Waiting for services to start..."
sleep 5
# Register runner
echo ""
echo -e "${YELLOW}Registering runner...${NC}"
docker compose exec -T gitea-runner act_runner register \
--instance "${GITEA_INSTANCE_URL}" \
--token "${GITEA_RUNNER_REGISTRATION_TOKEN}" \
--name "${GITEA_RUNNER_NAME:-dev-runner}" \
--labels "${GITEA_RUNNER_LABELS:-ubuntu-latest:docker://node:16-bullseye}"
if [ $? -eq 0 ]; then
echo ""
echo -e "${GREEN}✓ Runner registered successfully!${NC}"
echo ""
echo "Runner is now active and will start accepting jobs."
echo ""
echo "Useful commands:"
echo " docker compose logs -f gitea-runner # View runner logs"
echo " docker compose restart gitea-runner # Restart runner"
echo " docker compose down # Stop runner"
echo ""
echo "Check runner status in Gitea:"
echo " ${GITEA_INSTANCE_URL}/admin/actions/runners"
echo ""
else
echo ""
echo -e "${RED}✗ Registration failed${NC}"
echo "Check the logs for details:"
echo " docker compose logs gitea-runner"
echo ""
exit 1
fi

View File

@@ -0,0 +1,59 @@
#!/usr/bin/env bash
set -euo pipefail
# Gitea Actions Runner Unregistration Script
# This script unregisters the runner from the Gitea instance
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "$SCRIPT_DIR"
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
echo -e "${YELLOW}Gitea Actions Runner Unregistration${NC}"
echo "========================================"
# Check if runner is registered
if [ ! -f ./data/.runner ]; then
echo -e "${YELLOW}Warning: No registration file found${NC}"
echo "Runner may not be registered"
exit 0
fi
echo ""
read -p "Are you sure you want to unregister this runner? (y/N): " -n 1 -r
echo ""
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
echo "Unregistration cancelled"
exit 0
fi
# Stop services
echo ""
echo -e "${YELLOW}Stopping services...${NC}"
docker compose down
# Remove registration file
echo "Removing registration file..."
rm -f ./data/.runner
# Remove runner data
echo ""
read -p "Do you want to remove all runner data? (y/N): " -n 1 -r
echo ""
if [[ $REPLY =~ ^[Yy]$ ]]; then
echo "Removing runner data..."
rm -rf ./data/*
echo -e "${GREEN}✓ Runner data removed${NC}"
fi
echo ""
echo -e "${GREEN}✓ Runner unregistered successfully!${NC}"
echo ""
echo "Note: You may need to manually remove the runner from Gitea UI:"
echo " Go to: Site Administration > Actions > Runners"
echo " Find the runner and click 'Delete'"
echo ""