- Change stacks_base_path_default from /home/deploy to /home/deploy/deployment/stacks
- Matches actual server directory structure where stacks are located
- Check if destination directory exists separately from git repo check
- Remove directory if it exists but is not a git repository
- Prevents 'destination path already exists' error during clone
- ansible.builtin.git no longer supports owner and group parameters
- Set ownership in separate file task after git operations
- Fixes 'Unsupported parameters' error
- Use git_repo_url instead of git_repository_url in tasks
- Set git_repo_url based on whether git_repository_url is provided
- This completely avoids the recursive loop issue
- Change git_repository_url to use git_repository_url_default instead of self-reference
- Fixes 'Recursive loop detected in template' error in Ansible playbook
- Test commit to verify that workflow can now:
- Use php-ci image with Ansible
- Use ANSIBLE_VAULT_PASSWORD secret for vault decryption
- Successfully deploy to staging
- Change deploy-staging and deploy-production to use php-ci runner
- php-ci image has Ansible pre-installed, fixing 'ansible-playbook: command not found' error
- Test commit to verify that workflow can now:
- Use php-ci image from docker-dind
- Login to registry with configured secrets
- Build and push images successfully
- Add build-ci-image-production.sh script for building CI images on production
- Add BUILD_ON_PRODUCTION.md documentation
- Fix Dockerfile to handle optional PECL extensions for PHP 8.5 RC
This fixes the issue where Gitea workflows fail with:
'Error response from daemon: pull access denied for php-ci'
- Add gitea-service.yml with proper timeout configuration
- Service definition required for Traefik to route to Gitea
- Replaces old gitea.yml file that was removed
- Change Traefik local HTTP port from 8080 to 8081 (conflict with cadvisor)
- Change Traefik dashboard port to 8093 (conflicts with cadvisor, Hyperion)
- Update Gitea SSH service IP from 172.23.0.2 to 172.23.0.3
- Note: Gitea SSH works directly via Docker port mapping in local dev
- Traefik TCP routing only needed for production (host network mode)
- Add TCP entrypoint 'gitea-ssh' on port 2222 in static config
- Create TCP router configuration for routing SSH traffic to Gitea
- Use Gitea container IP (172.23.0.2) since Traefik runs in host network mode
- Routes git.michaelschiemer.de:2222 through Traefik instead of direct VPN access
- Add docker-compose-direct-access.yml for VPN-only admin access
- Configure Portainer on port 9002 (avoid MinIO conflict)
- Add grafana.ini to disable external plugin update checks
- Bind services to 10.8.0.1 (WireGuard VPN gateway)
This configuration enables direct access to admin services via WireGuard VPN
while removing Traefik routing overhead. Services are bound exclusively to
the VPN gateway IP to prevent public access.
The Redis container was failing with 'Permission denied' when trying to create
the appendonlydir for AOF (Append-Only File) persistence. The error occurred because:
1. Redis runs as root to read Docker Secrets from /run/secrets/redis_password
2. The /data volume is owned by UID 999 (default redis user)
3. cap_drop: ALL removed the CHOWN capability needed to create subdirectories
4. AOF persistence requires creating appendonlydir in /data with proper ownership
Solution:
- Added CHOWN capability: Allows Redis to create directories with correct ownership
- Added DAC_OVERRIDE capability: Allows writing to volume owned by different user
- Maintains all other security restrictions (no-new-privileges, minimal capabilities)
This fixes the continuous restart loop that persisted through commits:
- 5f7ebd9: Fixed healthcheck variable syntax
- 700fe81: Fixed entrypoint script variables
- bfe6a96: Changed healthcheck to read secret directly
The real issue was not the healthcheck but the permission error that prevented
Redis from starting in the first place.
Refs: Redis container logs showed:
'Can't open or create append-only dir appendonlydir: Permission denied'
The health check now reads the password directly from /run/secrets/redis_password
instead of relying on an environment variable, which is not available in the
health check context.
This resolves the 'container application-redis-1 is unhealthy' error.
Previous fix (5f7ebd9) only updated health check line but missed entrypoint script.
The entrypoint script was still using $$REDIS_PASSWORD (Docker Compose escaping)
instead of $REDIS_PASSWORD (shell variable syntax).
Changes:
- Line 180: export REDIS_PASSWORD=$(cat ...) - now uses single $
- Line 182: if [ -n "$REDIS_PASSWORD" ] - now uses single $
- Line 190: --requirepass "$REDIS_PASSWORD" - now uses single $
Technical explanation:
The command: block is a multi-line shell script passed to /bin/sh -c.
Within this shell script context, we use normal shell variable syntax with
single $ for variable references. The export statement makes REDIS_PASSWORD
available to both the Redis process and the health check command.
This completes the fix for: "container application-redis-1 is unhealthy"
Related: 5f7ebd9 (health check fix), b1e3a00 (fallback strategy)
- Changed health check from $$REDIS_PASSWORD to $REDIS_PASSWORD
- Double dollar sign is Docker Compose variable escaping (wrong context)
- Single dollar sign correctly references environment variable exported by entrypoint
- Health check runs in container shell where REDIS_PASSWORD is available
- Fixes 'container application-redis-1 is unhealthy' deployment failure
Changes:
- Export REDIS_PASSWORD from Docker Secret in entrypoint script
- Health check now uses exported environment variable instead of reading Secret file
- Increased start_period to 30s to allow more time for initialization
Why this works:
- Environment variables are accessible to both main process and health checks
- Docker Secret file reading in health check context was unreliable
- Export makes password available in same shell session for health check
Security:
- Password still sourced from Docker Secret (encrypted at rest)
- Only exported within container environment (not exposed externally)
- Redis still requires password authentication (--requirepass)
Deployment fix#11 (continued): Redis container health check
Changed health check to try without password first, then with Docker Secret.
This handles both scenarios where password might not be immediately available
or where the Secret read might fail in health check context.
Changes:
- Use CMD-SHELL instead of CMD for shell expansion support
- Try 'redis-cli ping' first (no auth)
- Fallback to authenticated ping if first attempt fails
- Properly quote password from Docker Secret
This is the eleventh cumulative fix for production deployment pipeline.
Related: commit 477fe67 (initial Redis health check fix)
User specified that all environments (local, staging, production) should
use external PostgreSQL stacks consistently instead of embedded database.
Changes:
- Removed db service definition from base config (lines 87-114)
- Removed db dependency from queue-worker service
- Updated php-test DB_HOST to use external 'postgres' service
This eliminates the need for production overrides and creates uniform
architecture across all environments. The application-db-1 container
will no longer be started, fixing deployment failure.
This is the tenth cumulative fix for production deployment pipeline.
Related commits:
- f97863a: Add image references to production config
- 5b5fdee: Fix registry upload with sequential push
- a1b9a53: Remove CI test file
- 0b54086: Fix Node.js cache dependency
- 6263d7a: Trigger CI workflow verification
- 08f6f64: Stable IMAGE_TAG
- 2e539ed: Add build: null overrides
- 6e1faab: Deploy docker-compose via scp
- c1d6a71: Fix scp working directory
- a4ca6e9: Trigger pipeline
- 0c0c3ba: Fix .env mount conflict
- 41882da: Fix storage/var volume mounts
- 477fe67: Fix Redis health check
Previous health check used incorrect command: redis-cli --raw incr ping
This increments a counter instead of checking Redis health.
Changed to proper health check:
- Use standard redis-cli ping command
- Authenticate with password from Docker Secret
- Verify PONG response with grep
This is the ninth cumulative fix for production deployment pipeline.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Docker named volumes cannot create mount points inside read-only directories.
Previous configuration attempted to mount storage and var-data volumes at subdirectories
inside a read-only base mount (/var/www/html:ro), causing deployment failures.
Changes:
- php service: Changed /var/www/html mount from :ro to :rw, removed storage volume
- queue-worker service: Changed mount to :rw, removed storage and var-data volumes
- scheduler service: Changed mount to :rw, removed storage and var-data volumes
Security maintained through:
- Container runs as non-root user (appuser via gosu)
- Security hardening (no-new-privileges, dropped capabilities)
- Rsync deployment from trusted source
This is the eighth cumulative fix for production deployment pipeline.
Remove separate .env file mounts from php, queue-worker, and scheduler
services to fix read-only filesystem mount conflict.
The .env file is already included in the rsync deployment at
/home/deploy/michaelschiemer/current/.env and is accessible through
the main application code mount. Separate file mounts are redundant
and cause Docker mount conflicts because they attempt to create mount
points inside read-only parent directories.
Error fixed:
- error mounting '/var/www/html/.env': read-only file system
Services fixed:
- php: removed .env mount (line 154)
- queue-worker: removed .env mount (line 254)
- scheduler: removed .env mount (line 327)
Update comment to trigger CI/CD workflow with cd /workspace/repo fix
(commit c1d6a71).
This deployment will:
1. cd to /workspace/repo before scp
2. Transfer docker-compose files successfully
3. Deploy all six fixes to production
The 'Deploy to Production Server' step is separate from the clone step and
doesn't inherit the working directory. The scp command was running from the
default directory where docker-compose files don't exist.
Adding 'cd /workspace/repo' before scp to access the cloned repository files.
This fixes the error:
docker-compose.base.yml: No such file or directory
docker-compose.production.yml: No such file or directory
Update comment to trigger CI/CD workflow with docker-compose file deployment
via scp (commit 6e1faab).
This deployment will:
1. Build and push image with stable git-SHA tag
2. Transfer docker-compose.base.yml and docker-compose.production.yml via scp
3. Pull new image on production server
4. Restart services with updated configuration including build: null overrides
All five deployment fixes are now integrated:
- Sequential push (0b342c6)
- Stable IMAGE_TAG (08f6f64)
- build: null overrides (2e539ed)
- scp docker-compose deployment (6e1faab)
- Source file trigger (this commit)
The previous 'always sync' approach failed because it tried to copy files
from /workspace/repo/ which doesn't exist on the production server.
The SSH heredoc (<<EOF) executes commands ON the production server, not in
the Gitea Actions workspace. File paths inside heredoc are relative to the
production server's filesystem.
This commit adds an scp step BEFORE the SSH heredoc to transfer
docker-compose.base.yml and docker-compose.production.yml from the Actions
workspace to the production server.
This ensures the build: null overrides (commit 2e539ed) reach production
and services can restart without build context errors.
Changes:
- Added scp command to deploy docker-compose files before SSH deployment
- Changed file sync check from 'cp' to file existence validation
- Updated comments to clarify rsync-based deployment architecture
Related commits:
- 0b342c6: Sequential push strategy
- 08f6f64: Stable git-SHA IMAGE_TAG
- 2e539ed: build: null overrides
- 0db73df: Always-sync docker-compose (incorrect implementation)
- 3091205: Trigger pipeline with source file change
The deployment script was only copying docker-compose files when missing,
preventing configuration updates (like build: null overrides) from being deployed.
Changed from conditional copy to always sync latest files from repository,
ensuring all configuration changes are properly deployed to production.
