The Redis container was failing with 'Permission denied' when trying to create
the appendonlydir for AOF (Append-Only File) persistence. The error occurred because:
1. Redis runs as root to read Docker Secrets from /run/secrets/redis_password
2. The /data volume is owned by UID 999 (default redis user)
3. cap_drop: ALL removed the CHOWN capability needed to create subdirectories
4. AOF persistence requires creating appendonlydir in /data with proper ownership
Solution:
- Added CHOWN capability: Allows Redis to create directories with correct ownership
- Added DAC_OVERRIDE capability: Allows writing to volume owned by different user
- Maintains all other security restrictions (no-new-privileges, minimal capabilities)
This fixes the continuous restart loop that persisted through commits:
- 5f7ebd9: Fixed healthcheck variable syntax
- 700fe81: Fixed entrypoint script variables
- bfe6a96: Changed healthcheck to read secret directly
The real issue was not the healthcheck but the permission error that prevented
Redis from starting in the first place.
Refs: Redis container logs showed:
'Can't open or create append-only dir appendonlydir: Permission denied'
The health check now reads the password directly from /run/secrets/redis_password
instead of relying on an environment variable, which is not available in the
health check context.
This resolves the 'container application-redis-1 is unhealthy' error.
Previous fix (5f7ebd9) only updated health check line but missed entrypoint script.
The entrypoint script was still using $$REDIS_PASSWORD (Docker Compose escaping)
instead of $REDIS_PASSWORD (shell variable syntax).
Changes:
- Line 180: export REDIS_PASSWORD=$(cat ...) - now uses single $
- Line 182: if [ -n "$REDIS_PASSWORD" ] - now uses single $
- Line 190: --requirepass "$REDIS_PASSWORD" - now uses single $
Technical explanation:
The command: block is a multi-line shell script passed to /bin/sh -c.
Within this shell script context, we use normal shell variable syntax with
single $ for variable references. The export statement makes REDIS_PASSWORD
available to both the Redis process and the health check command.
This completes the fix for: "container application-redis-1 is unhealthy"
Related: 5f7ebd9 (health check fix), b1e3a00 (fallback strategy)
- Changed health check from $$REDIS_PASSWORD to $REDIS_PASSWORD
- Double dollar sign is Docker Compose variable escaping (wrong context)
- Single dollar sign correctly references environment variable exported by entrypoint
- Health check runs in container shell where REDIS_PASSWORD is available
- Fixes 'container application-redis-1 is unhealthy' deployment failure
Changes:
- Export REDIS_PASSWORD from Docker Secret in entrypoint script
- Health check now uses exported environment variable instead of reading Secret file
- Increased start_period to 30s to allow more time for initialization
Why this works:
- Environment variables are accessible to both main process and health checks
- Docker Secret file reading in health check context was unreliable
- Export makes password available in same shell session for health check
Security:
- Password still sourced from Docker Secret (encrypted at rest)
- Only exported within container environment (not exposed externally)
- Redis still requires password authentication (--requirepass)
Deployment fix#11 (continued): Redis container health check
Changed health check to try without password first, then with Docker Secret.
This handles both scenarios where password might not be immediately available
or where the Secret read might fail in health check context.
Changes:
- Use CMD-SHELL instead of CMD for shell expansion support
- Try 'redis-cli ping' first (no auth)
- Fallback to authenticated ping if first attempt fails
- Properly quote password from Docker Secret
This is the eleventh cumulative fix for production deployment pipeline.
Related: commit 477fe67 (initial Redis health check fix)
User specified that all environments (local, staging, production) should
use external PostgreSQL stacks consistently instead of embedded database.
Changes:
- Removed db service definition from base config (lines 87-114)
- Removed db dependency from queue-worker service
- Updated php-test DB_HOST to use external 'postgres' service
This eliminates the need for production overrides and creates uniform
architecture across all environments. The application-db-1 container
will no longer be started, fixing deployment failure.
This is the tenth cumulative fix for production deployment pipeline.
Related commits:
- f97863a: Add image references to production config
- 5b5fdee: Fix registry upload with sequential push
- a1b9a53: Remove CI test file
- 0b54086: Fix Node.js cache dependency
- 6263d7a: Trigger CI workflow verification
- 08f6f64: Stable IMAGE_TAG
- 2e539ed: Add build: null overrides
- 6e1faab: Deploy docker-compose via scp
- c1d6a71: Fix scp working directory
- a4ca6e9: Trigger pipeline
- 0c0c3ba: Fix .env mount conflict
- 41882da: Fix storage/var volume mounts
- 477fe67: Fix Redis health check
Previous health check used incorrect command: redis-cli --raw incr ping
This increments a counter instead of checking Redis health.
Changed to proper health check:
- Use standard redis-cli ping command
- Authenticate with password from Docker Secret
- Verify PONG response with grep
This is the ninth cumulative fix for production deployment pipeline.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Docker named volumes cannot create mount points inside read-only directories.
Previous configuration attempted to mount storage and var-data volumes at subdirectories
inside a read-only base mount (/var/www/html:ro), causing deployment failures.
Changes:
- php service: Changed /var/www/html mount from :ro to :rw, removed storage volume
- queue-worker service: Changed mount to :rw, removed storage and var-data volumes
- scheduler service: Changed mount to :rw, removed storage and var-data volumes
Security maintained through:
- Container runs as non-root user (appuser via gosu)
- Security hardening (no-new-privileges, dropped capabilities)
- Rsync deployment from trusted source
This is the eighth cumulative fix for production deployment pipeline.
Remove separate .env file mounts from php, queue-worker, and scheduler
services to fix read-only filesystem mount conflict.
The .env file is already included in the rsync deployment at
/home/deploy/michaelschiemer/current/.env and is accessible through
the main application code mount. Separate file mounts are redundant
and cause Docker mount conflicts because they attempt to create mount
points inside read-only parent directories.
Error fixed:
- error mounting '/var/www/html/.env': read-only file system
Services fixed:
- php: removed .env mount (line 154)
- queue-worker: removed .env mount (line 254)
- scheduler: removed .env mount (line 327)
Update comment to trigger CI/CD workflow with cd /workspace/repo fix
(commit c1d6a71).
This deployment will:
1. cd to /workspace/repo before scp
2. Transfer docker-compose files successfully
3. Deploy all six fixes to production
The 'Deploy to Production Server' step is separate from the clone step and
doesn't inherit the working directory. The scp command was running from the
default directory where docker-compose files don't exist.
Adding 'cd /workspace/repo' before scp to access the cloned repository files.
This fixes the error:
docker-compose.base.yml: No such file or directory
docker-compose.production.yml: No such file or directory
Update comment to trigger CI/CD workflow with docker-compose file deployment
via scp (commit 6e1faab).
This deployment will:
1. Build and push image with stable git-SHA tag
2. Transfer docker-compose.base.yml and docker-compose.production.yml via scp
3. Pull new image on production server
4. Restart services with updated configuration including build: null overrides
All five deployment fixes are now integrated:
- Sequential push (0b342c6)
- Stable IMAGE_TAG (08f6f64)
- build: null overrides (2e539ed)
- scp docker-compose deployment (6e1faab)
- Source file trigger (this commit)
The previous 'always sync' approach failed because it tried to copy files
from /workspace/repo/ which doesn't exist on the production server.
The SSH heredoc (<<EOF) executes commands ON the production server, not in
the Gitea Actions workspace. File paths inside heredoc are relative to the
production server's filesystem.
This commit adds an scp step BEFORE the SSH heredoc to transfer
docker-compose.base.yml and docker-compose.production.yml from the Actions
workspace to the production server.
This ensures the build: null overrides (commit 2e539ed) reach production
and services can restart without build context errors.
Changes:
- Added scp command to deploy docker-compose files before SSH deployment
- Changed file sync check from 'cp' to file existence validation
- Updated comments to clarify rsync-based deployment architecture
Related commits:
- 0b342c6: Sequential push strategy
- 08f6f64: Stable git-SHA IMAGE_TAG
- 2e539ed: build: null overrides
- 0db73df: Always-sync docker-compose (incorrect implementation)
- 3091205: Trigger pipeline with source file change
The deployment script was only copying docker-compose files when missing,
preventing configuration updates (like build: null overrides) from being deployed.
Changed from conditional copy to always sync latest files from repository,
ensuring all configuration changes are properly deployed to production.
Added 'build: null' to web, php, and queue-worker services in docker-compose.production.yml
to explicitly remove build sections inherited from base config.
This fixes 'lstat /home/deploy/deployment/stacks/application/docker/php: no such file or directory'
error during deployment, as production servers only have docker-compose files, not build context.
Registry-based deployment should pull pre-built images, not attempt to build on production server.
Changed IMAGE_TAG output from timestamp-based format to stable git-SHA format to ensure
deployment can reliably pull the image that was actually pushed to the registry.
Before: IMAGE_TAG="6c7040e-1762265632" (changes with time)
After: IMAGE_TAG="git-6c7040e" (stable, matches pushed tag)
This fixes deployment manifest not found errors.
Docker registry was getting overwhelmed with concurrent pushes of
multiple tags and cache layers, resulting in 499 status code
(Client Closed Request).
Changes:
- Build with --load instead of --push to save image locally first
- Push each tag sequentially (latest, timestamp, git-sha) instead of all at once
- Reduce cache targets from 2 to 1 (keep only buildcache)
- Add progress logging for each push operation
This approach:
1. Reduces concurrent write pressure on registry
2. Allows better error handling per tag
3. Provides clearer progress feedback
4. Prevents registry timeouts from concurrent uploads
Related to: Status 499 error during docker push
Production deployment was failing because docker-compose.production.yml
had build: sections but no image: references. This caused Docker Compose
to attempt building on the server, which failed because the docker/
directory doesn't exist in the deployment location.
Changes:
- Add image: git.michaelschiemer.de:5000/framework:latest to web, php, and queue-worker services
- Removed build: section from php service (no longer needed)
- Remove test comment from ShowHome.php
The deployment script's sed command (line 1259-1260 in build-image.yml)
now successfully finds and updates the image: tags with the correct
version from the registry.
Related to: Production deployment error "docker/php: no such file or directory"
- Add example secret files for `app_key`, `db_user_password`, and `redis_password`.
- Introduce `local.vault.yml.example` for Ansible Vault encryption of local secrets.
- Create migration and setup scripts for transitioning from `.env.local` to secrets files.
- Update `docker-compose.local.yml` to adopt Docker Secrets and `_FILE` pattern for local configurations.
- Add deployment playbooks and enhanced logging configurations for local development.
- Mark `RedisConnection::$connected` as read-only with `private(set)`.
- Simplify authentication and database selection logic in `RedisConnection`.
- Comment out DI container singleton registration in `RedisPoolInitializer`.
- Annotate `RedisConnectionPool` with `#[Singleton]` attribute for improved clarity.
- Remove redundant fallback for `RedisConfig` key prefix to enforce explicit configuration.
- Refine `ClassExtractor` with class name validation to exclude invalid identifiers and handle creation errors.
- Improve `AttributeCache` by validating class existence before reflection, preventing unnecessary exceptions and caching empty results on failure.
- Remove default values for `RedisConfig` constructor to enforce explicit configuration.
- Enhance `FileStreamProcessor` logging by adding `LogContext` with exception details.
- Replace `humanReadable` method call with `toHumanReadable` in `DiscoveryCompletedEvent`.
- Remove redundant error trace logging in `CacheInitializer` for cleaner fallback handling.
- Introduce `normalizeTypeName` to validate and normalize type names during dependency analysis.
- Add `safeCreateClassName` to handle `ClassName` creation errors gracefully.
- Enhance constructor, method, property, and return type dependency edge creation with context-aware namespace resolution.
- Improve logging to capture failure details and provide debugging insights.
- Add `FileOwnership` to encapsulate file owner and group information.
- Add `ProcessUser` to represent and manage system process user details.
- Enhance ownership matching and debugging with structured data objects.
- Include new documentation on file ownership handling and permission improvements.
- Prepare infrastructure for enriched error handling in filesystem operations.
- Introduce `cache-configuration.md` for detailed instructions on cache setup, permission troubleshooting, and best practices.
- Add `cache-permissions-quick-fix.md` for concise resolutions to common permission errors.
- Include a detailed `FILECACHE_PERMISSION_FIX_PLAN.md` outlining solutions for permission-related issues.
- Enhance `docker-entrypoint.sh` with permission fixes for multi-user caches.
- Update `Makefile` with cache clear commands for local and staging environments.
- Improve `FileCache` for graceful degradation on permission errors, ensuring reliability under multi-user scenarios.
- Introduce `EnumResolver` to centralize and cache enum value conversions.
- Enhance `DockerSecretsResolver` with result caching to avoid redundant file reads and improve performance.
- Update `Environment` to integrate `EnumResolver` for enriched enum resolution support and improved maintainability.
- Adjust unit tests to validate caching mechanisms and error handling improvements.
- Refactor `FileCache` methods to enhance file operation consistency and error handling.
- Integrate `LockableStorage` for improved locking with fallback to manual lock implementations.
- Replace `glob` usage with `FileSystem` module for directory operations, improving maintainability and testability.
- Optimize cache file listing, filtering, and expiration handling for better performance and reliability.
- Streamline directory and file deletion logic with additional error resilience.
- Replace `DefaultContainer` lookups with direct constructor injection in `TemplateProcessorInitializer` and `TemplateRendererInitializer`.
- Streamline method logic by removing redundant operations and ensuring dependencies are passed explicitly.
- Enhance readability and maintainability by reducing unnecessary indirections.
- Refine branch and path filters for efficient CI triggers.
- Add concurrency control to avoid overlapping builds.
- Improve runtime base build logic with conditional evaluation and skipping.
- Enhance image info generation with fallback handling and deployment readiness checks.
- Introduce `CodeParser` to extract dependencies from `container->get()` calls and `return new` statements.
- Add `DependencyPathAnalyzer` for recursive analysis of dependency paths with cycle detection.
- Implement `InitializerFinder` to locate initializers based on naming conventions.
- Include `InterfaceResolver` to determine interface implementations using introspection and initializers.
- Add `NamespaceResolver` for resolving class names from use statements and namespaces.
- Introduce `ReturnTypeAnalyzer` for method and closure return type analysis.
