The Redis container was failing with 'Permission denied' when trying to create
the appendonlydir for AOF (Append-Only File) persistence. The error occurred because:
1. Redis runs as root to read Docker Secrets from /run/secrets/redis_password
2. The /data volume is owned by UID 999 (default redis user)
3. cap_drop: ALL removed the CHOWN capability needed to create subdirectories
4. AOF persistence requires creating appendonlydir in /data with proper ownership
Solution:
- Added CHOWN capability: Allows Redis to create directories with correct ownership
- Added DAC_OVERRIDE capability: Allows writing to volume owned by different user
- Maintains all other security restrictions (no-new-privileges, minimal capabilities)
This fixes the continuous restart loop that persisted through commits:
- 5f7ebd9: Fixed healthcheck variable syntax
- 700fe81: Fixed entrypoint script variables
- bfe6a96: Changed healthcheck to read secret directly
The real issue was not the healthcheck but the permission error that prevented
Redis from starting in the first place.
Refs: Redis container logs showed:
'Can't open or create append-only dir appendonlydir: Permission denied'
The health check now reads the password directly from /run/secrets/redis_password
instead of relying on an environment variable, which is not available in the
health check context.
This resolves the 'container application-redis-1 is unhealthy' error.
Previous fix (5f7ebd9) only updated health check line but missed entrypoint script.
The entrypoint script was still using $$REDIS_PASSWORD (Docker Compose escaping)
instead of $REDIS_PASSWORD (shell variable syntax).
Changes:
- Line 180: export REDIS_PASSWORD=$(cat ...) - now uses single $
- Line 182: if [ -n "$REDIS_PASSWORD" ] - now uses single $
- Line 190: --requirepass "$REDIS_PASSWORD" - now uses single $
Technical explanation:
The command: block is a multi-line shell script passed to /bin/sh -c.
Within this shell script context, we use normal shell variable syntax with
single $ for variable references. The export statement makes REDIS_PASSWORD
available to both the Redis process and the health check command.
This completes the fix for: "container application-redis-1 is unhealthy"
Related: 5f7ebd9 (health check fix), b1e3a00 (fallback strategy)
- Changed health check from $$REDIS_PASSWORD to $REDIS_PASSWORD
- Double dollar sign is Docker Compose variable escaping (wrong context)
- Single dollar sign correctly references environment variable exported by entrypoint
- Health check runs in container shell where REDIS_PASSWORD is available
- Fixes 'container application-redis-1 is unhealthy' deployment failure
Changes:
- Export REDIS_PASSWORD from Docker Secret in entrypoint script
- Health check now uses exported environment variable instead of reading Secret file
- Increased start_period to 30s to allow more time for initialization
Why this works:
- Environment variables are accessible to both main process and health checks
- Docker Secret file reading in health check context was unreliable
- Export makes password available in same shell session for health check
Security:
- Password still sourced from Docker Secret (encrypted at rest)
- Only exported within container environment (not exposed externally)
- Redis still requires password authentication (--requirepass)
Deployment fix#11 (continued): Redis container health check
Changed health check to try without password first, then with Docker Secret.
This handles both scenarios where password might not be immediately available
or where the Secret read might fail in health check context.
Changes:
- Use CMD-SHELL instead of CMD for shell expansion support
- Try 'redis-cli ping' first (no auth)
- Fallback to authenticated ping if first attempt fails
- Properly quote password from Docker Secret
This is the eleventh cumulative fix for production deployment pipeline.
Related: commit 477fe67 (initial Redis health check fix)
Previous health check used incorrect command: redis-cli --raw incr ping
This increments a counter instead of checking Redis health.
Changed to proper health check:
- Use standard redis-cli ping command
- Authenticate with password from Docker Secret
- Verify PONG response with grep
This is the ninth cumulative fix for production deployment pipeline.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Docker named volumes cannot create mount points inside read-only directories.
Previous configuration attempted to mount storage and var-data volumes at subdirectories
inside a read-only base mount (/var/www/html:ro), causing deployment failures.
Changes:
- php service: Changed /var/www/html mount from :ro to :rw, removed storage volume
- queue-worker service: Changed mount to :rw, removed storage and var-data volumes
- scheduler service: Changed mount to :rw, removed storage and var-data volumes
Security maintained through:
- Container runs as non-root user (appuser via gosu)
- Security hardening (no-new-privileges, dropped capabilities)
- Rsync deployment from trusted source
This is the eighth cumulative fix for production deployment pipeline.
Remove separate .env file mounts from php, queue-worker, and scheduler
services to fix read-only filesystem mount conflict.
The .env file is already included in the rsync deployment at
/home/deploy/michaelschiemer/current/.env and is accessible through
the main application code mount. Separate file mounts are redundant
and cause Docker mount conflicts because they attempt to create mount
points inside read-only parent directories.
Error fixed:
- error mounting '/var/www/html/.env': read-only file system
Services fixed:
- php: removed .env mount (line 154)
- queue-worker: removed .env mount (line 254)
- scheduler: removed .env mount (line 327)
Added 'build: null' to web, php, and queue-worker services in docker-compose.production.yml
to explicitly remove build sections inherited from base config.
This fixes 'lstat /home/deploy/deployment/stacks/application/docker/php: no such file or directory'
error during deployment, as production servers only have docker-compose files, not build context.
Registry-based deployment should pull pre-built images, not attempt to build on production server.
Production deployment was failing because docker-compose.production.yml
had build: sections but no image: references. This caused Docker Compose
to attempt building on the server, which failed because the docker/
directory doesn't exist in the deployment location.
Changes:
- Add image: git.michaelschiemer.de:5000/framework:latest to web, php, and queue-worker services
- Removed build: section from php service (no longer needed)
- Remove test comment from ShowHome.php
The deployment script's sed command (line 1259-1260 in build-image.yml)
now successfully finds and updates the image: tags with the correct
version from the registry.
Related to: Production deployment error "docker/php: no such file or directory"
- Mount /home/deploy/michaelschiemer/current:/var/www/html:ro in php and queue-worker services
- This allows deployment via rsync without requiring Docker image rebuild
- Storage volume still mounted as writable overlay for runtime data
- Change default DB_DRIVER to 'pgsql' for PostgreSQL
Deployment Architecture:
- rsync deploys code to /home/deploy/michaelschiemer/releases/{timestamp}
- Atomic symlink switch to /home/deploy/michaelschiemer/current
- PHP containers mount current/ for immediate code updates
- No rebuild needed - code changes are live after symlink switch
Benefits:
- Faster deployments (no Docker rebuild)
- Code changes reflected immediately
- Zero-downtime releases
- Easy rollback via symlink change
- Create AnsibleDeployStage using framework's Process module for secure command execution
- Integrate AnsibleDeployStage into DeploymentPipelineCommands for production deployments
- Add force_deploy flag support in Ansible playbook to override stale locks
- Use PHP deployment module as orchestrator (php console.php deploy:production)
- Fix ErrorAggregationInitializer to use Environment class instead of $_ENV superglobal
Architecture:
- BuildStage → AnsibleDeployStage → HealthCheckStage for production
- Process module provides timeout, error handling, and output capture
- Ansible playbook supports rollback via rollback-git-based.yml
- Zero-downtime deployments with health checks
