michaelschiemer/deployment/stacks/gitea/README.md

# Gitea Stack - Self-Hosted Git Server

## Overview

Gitea acts as the central Git server with integrated CI/CD capabilities through Gitea Actions, handling:
- Git repository hosting
- User and organization management
- Pull requests and code reviews
- Issue tracking
- Gitea Actions for CI/CD (runner runs on development machine)
- API for automation

## Services

- **git.michaelschiemer.de** - Gitea Web Interface
- **git.michaelschiemer.de:2222** - SSH for Git operations
- **MySQL 8.0** - Database backend
- **Redis 7** - Cache, session, and queue storage

## Prerequisites

1. **Traefik Stack Running**
   ```bash
   cd ../traefik
   docker compose up -d
   ```

2. **DNS Configuration**
   Point `git.michaelschiemer.de` to your server IP (94.16.110.151)

3. **SSH Port Availability**
   Ensure port 2222 is open in your firewall for Git SSH operations

## Configuration

### 1. Create Environment File

```bash
cp .env.example .env
```

### 2. Generate Strong Passwords

```bash
# MySQL root password
openssl rand -base64 32

# MySQL gitea password
openssl rand -base64 32

# Redis password
openssl rand -base64 32
```

Update `.env` with generated passwords:
```env
MYSQL_ROOT_PASSWORD=<generated-password-1>
MYSQL_PASSWORD=<generated-password-2>
REDIS_PASSWORD=<generated-password-3>
```

### 3. Adjust Configuration (Optional)

Edit `.env` for:
- Domain customization
- User registration settings
- Database configuration

## Deployment

### Initial Setup

```bash
# Deploy stack
docker compose up -d

# Check logs
docker compose logs -f

# Wait for MySQL initialization (30-60 seconds)
docker compose logs mysql | grep "ready for connections"

# Verify services are healthy
docker compose ps
```

### First Time Configuration

**Option 1: Automated Setup (Recommended)**

The Gitea initial setup can be automated using Ansible:

```bash
cd deployment/ansible

# 1. Set Gitea admin credentials in vault
ansible-vault edit secrets/production.vault.yml --vault-password-file secrets/.vault_pass

# Add these variables:
# vault_gitea_admin_username: "admin"
# vault_gitea_admin_password: "your-secure-password"
# vault_gitea_admin_email: "kontakt@michaelschiemer.de"

# 2. Run the setup playbook
ansible-playbook -i inventory/production.yml \
  playbooks/setup-gitea-initial-config.yml \
  --vault-password-file secrets/.vault_pass
```

The playbook will:
- Check if Gitea is already configured
- Generate `app.ini` with `INSTALL_LOCK = true` to skip the initial setup page
- Copy the configuration file to the Gitea container
- Create admin user via Gitea CLI with credentials from vault
- Use database settings from environment variables

**How it works:**
The playbook creates a complete `app.ini` configuration file with `INSTALL_LOCK = true` in the `[security]` section. This tells Gitea to skip the initial setup wizard. The admin user is then created using the `gitea admin user create` command.

**Option 2: Manual Setup**

1. **Access Gitea**: https://git.michaelschiemer.de

2. **Initial Setup Wizard**:
   - Database settings are pre-configured via environment variables
   - Set up admin account:
     - Username: `admin` (or your preference)
     - Email: `kontakt@michaelschiemer.de`
     - Password: Strong password
   - Server and third-party settings: Use defaults
   - Click "Install Gitea"

3. **Verify SSH Access**:
   ```bash
   # Test SSH connection (replace 'git' with your username after setup)
   ssh -T -p 2222 git@git.michaelschiemer.de
   ```

## Usage

### Creating a Repository

**Option 1: Automated Setup (Recommended)**

Use Ansible to automatically create the repository and configure Git remote:

```bash
cd deployment/ansible

ansible-playbook -i inventory/production.yml \
  playbooks/setup-gitea-repository.yml \
  --vault-password-file secrets/.vault_pass \
  -e "repo_name=michaelschiemer" \
  -e "repo_owner=michael" \
  -e "repo_private=false"
```

The playbook will:
- Create repository in Gitea via API
- Configure Git remote automatically
- Use credentials from Ansible Vault

**Option 2: Manual Setup**

1. Log in to https://git.michaelschiemer.de
2. Click "+" → "New Repository"
3. Fill in repository details
4. Clone via HTTPS or SSH:
   ```bash
   # HTTPS
   git clone https://git.michaelschiemer.de/username/repo.git

   # SSH
   git clone ssh://git@git.michaelschiemer.de:2222/username/repo.git
   ```

### Configuration File

Gitea configuration is managed via `app.ini` file:
- **Template**: `deployment/ansible/templates/gitea-app.ini.j2` (Ansible template)
- **Production**: Generated from template and deployed via Ansible playbook `setup-gitea-initial-config.yml`
- The `app.ini` is copied to the container at `/data/gitea/conf/app.ini`
- **Important**: `app.ini` is a minimal configuration. Cache, Session, Queue, and other settings are controlled via `GITEA__...` environment variables in `docker-compose.yml` which override `app.ini` settings on every container start.
- Configuration is based on the official Gitea example: https://github.com/go-gitea/gitea/blob/main/custom/conf/app.example.ini

**Key Configuration Sections:**
- `[server]`: Domain, ports, SSH settings
- `[database]`: PostgreSQL connection
- `[actions]`: Actions enabled, no GitHub dependency
- `[service]`: Registration settings
- `[cache]` / `[session]` / `[queue]`: Storage configuration

### Gitea Actions

Gitea Actions (GitHub Actions compatible) are enabled by default. To use them:

1. **Create `.gitea/workflows/` directory** in your repository
2. **Add workflow YAML files** (e.g., `deploy.yml`)
3. **Register a Runner** (see Runner setup section below)

**Note**: The Gitea Actions Runner should run on your **development machine**, not on the production server. See Stack 9 documentation for runner setup.

### User Management

**Disable Registration** (Default):
- Set `DISABLE_REGISTRATION=true` in `.env` (already default)
- Create users via Admin Panel

**Enable Registration**:
- Set `DISABLE_REGISTRATION=false` in `.env`
- Restart: `docker compose restart gitea`

### Organizations and Teams

1. Navigate to Organizations
2. Create organization
3. Add repositories to organization
4. Manage teams and permissions

## API Access

Gitea provides a comprehensive API:

```bash
# Generate API token
# Settings → Applications → Generate New Token

# Example: List repositories
curl -H "Authorization: token YOUR_TOKEN" \
     https://git.michaelschiemer.de/api/v1/user/repos
```

**API Documentation**: https://git.michaelschiemer.de/api/swagger

## Backup & Recovery

### Manual Backup

```bash
# Backup script (run on production server)
#!/bin/bash
BACKUP_DIR="/backups/gitea"
DATE=$(date +%Y%m%d_%H%M%S)

# Create backup directory
mkdir -p $BACKUP_DIR

# Backup Gitea data
docker run --rm \
  -v gitea-data:/data \
  -v $BACKUP_DIR:/backup \
  alpine tar czf /backup/gitea-data-$DATE.tar.gz -C /data .

# Backup MySQL database
docker exec gitea-mysql mysqldump \
  -u root -p$MYSQL_ROOT_PASSWORD \
  --all-databases \
  --single-transaction \
  --quick \
  --lock-tables=false \
  > $BACKUP_DIR/gitea-mysql-$DATE.sql

# Backup Redis data
docker run --rm \
  -v gitea-redis-data:/data \
  -v $BACKUP_DIR:/backup \
  alpine tar czf /backup/gitea-redis-$DATE.tar.gz -C /data .

echo "Backup completed: $BACKUP_DIR/*-$DATE.*"
```

### Restore from Backup

```bash
# Stop services
docker compose down

# Restore Gitea data
docker run --rm \
  -v gitea-data:/data \
  -v /backups/gitea:/backup \
  alpine tar xzf /backup/gitea-data-YYYYMMDD_HHMMSS.tar.gz -C /data

# Restore MySQL
cat /backups/gitea/gitea-mysql-YYYYMMDD_HHMMSS.sql | \
  docker exec -i gitea-mysql mysql -u root -p$MYSQL_ROOT_PASSWORD

# Restore Redis
docker run --rm \
  -v gitea-redis-data:/data \
  -v /backups/gitea:/backup \
  alpine tar xzf /backup/gitea-redis-YYYYMMDD_HHMMSS.tar.gz -C /data

# Start services
docker compose up -d
```

### Automated Backups

Add to crontab on production server:

```bash
# Daily backup at 2 AM
0 2 * * * /path/to/backup-gitea.sh

# Keep only last 7 days
0 3 * * * find /backups/gitea -type f -mtime +7 -delete
```

## Monitoring

### Health Checks

```bash
# Check service health
docker compose ps

# Gitea health endpoint
curl -f https://git.michaelschiemer.de/api/healthz

# MySQL health
docker exec gitea-mysql mysqladmin ping -h localhost -u root -p$MYSQL_ROOT_PASSWORD

# Redis health
docker exec gitea-redis redis-cli -a $REDIS_PASSWORD ping
```

### Logs

```bash
# All services
docker compose logs -f

# Gitea only
docker compose logs -f gitea

# MySQL only
docker compose logs -f mysql

# Redis only
docker compose logs -f redis

# MySQL slow queries
docker exec gitea-mysql tail -f /var/log/mysql/slow-queries.log
```

### Resource Usage

```bash
# Container stats
docker stats gitea gitea-mysql gitea-redis

# Disk usage
docker system df -v | grep gitea
```

## Troubleshooting

### Gitea Not Starting

```bash
# Check logs
docker compose logs gitea

# Common issues:
# 1. MySQL not ready - wait 30-60 seconds
# 2. Database connection failed - check MYSQL_PASSWORD in .env
# 3. Redis connection failed - check REDIS_PASSWORD
```

### SSH Not Working

```bash
# Verify port 2222 is open
sudo ufw status | grep 2222

# Open if needed
sudo ufw allow 2222/tcp

# Test SSH connection
ssh -T -p 2222 git@git.michaelschiemer.de

# Check Gitea SSH settings
# Admin Panel → Configuration → Server and Other Services → SSH Server Domain
```

### Database Connection Issues

```bash
# Verify MySQL is running and healthy
docker compose ps mysql

# Test database connection
docker exec gitea-mysql mysql -u gitea -p$MYSQL_PASSWORD -e "SELECT 1;"

# Check MySQL logs
docker compose logs mysql | grep -i error
```

### Redis Connection Issues

```bash
# Verify Redis is running
docker compose ps redis

# Test Redis connection
docker exec gitea-redis redis-cli -a $REDIS_PASSWORD ping

# Check Redis logs
docker compose logs redis
```

### Performance Issues

**If Gitea has frequent outages or connection issues:**

1. **Update Gitea Configuration** (Recommended):
   ```bash
   cd deployment/ansible
   ansible-playbook -i inventory/production.yml \
     playbooks/update-gitea-config.yml \
     --vault-password-file secrets/.vault_pass
   ```
   
   This playbook will:
   - Enable Redis cache for better performance and persistence
   - Configure database connection pooling
   - Set connection limits to prevent "Connection reset by peer" errors

2. **Manual Troubleshooting**:
   ```bash
   # Check PostgreSQL slow queries
   docker exec gitea-postgres psql -U gitea -d gitea -c "SELECT * FROM pg_stat_activity;"
   
   # Check container resource usage
   docker stats gitea gitea-postgres gitea-redis
   
   # Check Gitea logs for errors
   docker compose logs --tail 100 gitea | grep -i error
   
   # Check Redis connection
   docker exec gitea-redis redis-cli -a $REDIS_PASSWORD ping
   ```

### Known Issues

**Bad Gateway after many rapid requests (15-20 reloads):**
- **Status**: Known issue, non-critical
- **Symptoms**: Gitea returns "Bad Gateway" after 15-20 rapid page reloads, recovers after a few seconds
- **Impact**: Low - Gitea is functional for normal usage
- **Possible causes**:
  - Container restart during high load
  - Connection pool exhaustion (mitigated with increased limits)
  - Traefik service discovery delay in host network mode
- **Workarounds**:
  - Wait a few seconds and retry
  - Use Redis cache (already enabled) for better performance
  - Consider adding rate limiting if needed (see Traefik middlewares)
- **Future improvements**:
  - Monitor and optimize connection pool usage
  - Consider adding rate limiting middleware for Gitea
  - Investigate Traefik service discovery in host network mode

### Reset Admin Password

```bash
# Connect to Gitea container
docker exec -it gitea bash

# Change admin password
gitea admin user change-password --username admin --password new-password
```

## Security

### Security Best Practices

1. **Disable User Registration**: Set `DISABLE_REGISTRATION=true`
2. **Strong Passwords**: Use generated passwords for all services
3. **Regular Updates**: Keep Gitea, MySQL, and Redis updated
4. **SSH Keys**: Prefer SSH keys over HTTPS for Git operations
5. **2FA**: Enable two-factor authentication for admin accounts
6. **API Token Security**: Rotate tokens regularly
7. **Firewall**: Only expose ports 80, 443, and 2222

### Update Stack

```bash
# Pull latest images
docker compose pull

# Recreate containers
docker compose up -d

# Verify
docker compose ps
```

### Security Headers

Security headers are applied via Traefik's `default-chain@file` middleware:
- HSTS
- Content-Type Nosniff
- XSS Protection
- Frame Deny
- CSP

## Integration with Other Stacks

### Docker Registry (Stack 3)

Gitea Actions can push built images to the private Docker Registry:

```yaml
# .gitea/workflows/deploy.yml
- name: Push to Registry
  run: |
    docker login registry.michaelschiemer.de -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }}
    docker push registry.michaelschiemer.de/myapp:latest
```

### Application Stack (Stack 4)

Deploy applications via Gitea Actions + Ansible:

```yaml
- name: Deploy to Production
  run: |
    ansible-playbook -i inventory/production deploy.yml
```

## Performance Tuning

### MySQL Optimization

Adjust `mysql/conf.d/gitea.cnf`:
- `innodb_buffer_pool_size`: Increase for more RAM
- `max_connections`: Increase for more concurrent users
- `slow_query_log`: Monitor slow queries

### Redis Optimization

```bash
# Add to docker-compose.yml redis command:
# --maxmemory 512mb --maxmemory-policy allkeys-lru
```

### Gitea Configuration

Edit via Admin Panel → Configuration or `app.ini`:
- Enable caching for static assets
- Adjust session timeout
- Configure queue workers for Actions

## Additional Resources

- **Gitea Documentation**: https://docs.gitea.io/
- **Gitea Actions**: https://docs.gitea.io/en-us/usage/actions/overview/
- **API Documentation**: https://git.michaelschiemer.de/api/swagger
- **MySQL Tuning**: https://dev.mysql.com/doc/refman/8.0/en/optimization.html