michaelschiemer/tests/e2e/livecomponents/CHUNKED-UPLOAD-TESTS.md

LiveComponents Chunked Upload E2E Tests

Comprehensive end-to-end testing suite for the LiveComponents chunked upload system.

Overview

This test suite validates the complete chunked upload functionality including:

• Upload session initialization and management
• Chunk splitting and parallel uploads
• SHA-256 integrity verification
• Progress tracking via SSE
• Resume capability for interrupted uploads
• Retry logic with exponential backoff
• Quarantine system integration
• Component state synchronization

Quick Start

Prerequisites

# Ensure Playwright is installed
npm install

# Install browser binaries
npx playwright install chromium

# Ensure development server is running
make up

Running Tests

# Run all chunked upload tests
npm run test:upload

# Run with visible browser (for debugging)
npm run test:upload:headed

# Run with debug mode
npm run test:upload:debug

# Run specific test
npx playwright test chunked-upload.spec.js --grep "should upload chunks in parallel"

Test Scenarios

1. Upload Session Initialization

Tests: Session creation with UUID format validation

Validates:

• Unique session ID generation
• UUID format (RFC 4122)
• Session metadata storage
• Client-server session sync

Expected Behavior:

• Session ID: ^[a-f0-9-]{36}$ (UUID v4 format)
• Session created within 500ms
• Session accessible via window.__uploadSession

2. Chunk Splitting Verification

Tests: File split into correct number of chunks based on size

Validates:

• Chunk size calculation (default: 512KB)
• Total chunks = ceil(fileSize / chunkSize)
• Chunk boundaries alignment

Example:

• 2MB file → 4 chunks (2048KB / 512KB = 4)
• 1MB file → 2 chunks (1024KB / 512KB = 2)
• 600KB file → 2 chunks (ceil(600KB / 512KB) = 2)

3. Parallel Chunk Uploads

Tests: Chunks upload concurrently (default: 3 parallel)

Validates:

• Multiple chunks upload simultaneously
• Concurrent request detection (time window analysis)
• Upload parallelization efficiency

Metrics:

• Expected concurrency: 3 simultaneous uploads
• Time window: Requests within 100ms considered parallel

4. Progress Tracking Accuracy

Tests: Real-time progress updates from 0% to 100%

Validates:

• Monotonic progress increase
• Accurate percentage calculation
• No progress regression
• Final state: 100% completion

Progress Formula:

progress = (uploadedChunks / totalChunks) * 100

5. SHA-256 Integrity Verification

Tests: Hash calculation and verification for each chunk

Validates:

• Client-side hash generation (SHA-256)
• Server-side hash verification
• Hash format: 64 hexadecimal characters
• Upload rejection on hash mismatch

Hash Calculation:

const hash = await crypto.subtle.digest('SHA-256', chunkData);
const hashHex = Array.from(new Uint8Array(hash))
    .map(b => b.toString(16).padStart(2, '0'))
    .join('');

6. Upload Interruption and Resume

Tests: Resume capability after page reload or connection loss

Validates:

• Uploaded chunks persistence
• Resume from last successful chunk
• Skip already uploaded chunks
• Session restoration

Resume Workflow:

1. Upload starts, some chunks complete
2. Interruption occurs (reload/disconnect)
3. Session restored with uploaded chunks info
4. Upload resumes from next chunk
5. Completion without re-uploading existing chunks

7. Retry Logic with Exponential Backoff

Tests: Failed chunks retry with increasing delays

Validates:

• Automatic retry on failure
• Exponential backoff strategy
• Maximum retry attempts (default: 3)
• Success after retries

Backoff Strategy:

delay = baseDelay * Math.pow(2, attemptNumber)
// Attempt 1: 100ms
// Attempt 2: 200ms
// Attempt 3: 400ms

8. Concurrent Multi-File Uploads

Tests: Multiple files upload simultaneously with queue management

Validates:

• Independent upload sessions per file
• Progress tracking per file
• Queue management (max concurrent files)
• No session conflicts

Expected Behavior:

• Each file: unique session ID
• Independent progress tracking
• Proper resource cleanup

9. Real-time SSE Progress Updates

Tests: Server-Sent Events for live progress updates

Validates:

• SSE connection establishment
• Progress events received
• Event format and data accuracy
• Connection persistence

SSE Event Format:

{
    type: 'upload-progress',
    sessionId: 'uuid',
    progress: 0-100,
    uploadedChunks: number,
    totalChunks: number,
    bytesUploaded: number,
    totalBytes: number
}

10. Upload Cancellation

Tests: User-initiated upload cancellation

Validates:

• Cancel button functionality
• Pending chunk requests abortion
• Session cleanup
• UI state reset

Cancellation Steps:

1. Upload in progress
2. Cancel button clicked
3. All pending requests aborted
4. Session marked as cancelled
5. Temporary chunks deleted

11. File Size Validation

Tests: File size limits enforcement

Validates:

• Maximum file size check (configurable)
• Client-side pre-validation
• Server-side validation
• Clear error messaging

Default Limits:

• Max file size: 100MB (configurable)
• Validation occurs before session creation

12. File Type Validation

Tests: Allowed file types enforcement

Validates:

• MIME type checking
• File extension validation
• Blocked file types rejection
• Whitelist/blacklist support

Example Allowed Types:

allowedTypes: [
    'image/jpeg', 'image/png', 'image/gif',
    'application/pdf', 'application/zip'
]

13. Uploaded File Display

Tests: File appears in component after successful upload

Validates:

• Fragment update with file list
• File metadata display (name, size, date)
• DOM update verification
• Component state consistency

Expected DOM:

<div data-lc-fragment="file-list">
    <div class="file-item" data-file-id="uuid">
        <span class="file-name">example.pdf</span>
        <span class="file-size">2.4 MB</span>
        <span class="file-date">2025-01-19</span>
    </div>
</div>

14. Quarantine System Integration

Tests: Uploaded files move to quarantine for virus scanning

Validates:

• File moved to quarantine directory
• Virus scan initiated (ClamAV/VirusTotal)
• Quarantine status tracking
• Clean/infected file handling

Quarantine Workflow:

1. Upload completes successfully
2. File moved to quarantine directory
3. Virus scan job queued
4. Component shows "Scanning..." status
5. Result: Clean → Available, Infected → Deleted

15. Component State Updates

Tests: Component state synchronizes after upload

Validates:

• State updates via LiveComponents protocol
• Fragment rendering with new data
• State persistence across interactions
• No state corruption

16. Network Interruption Handling

Tests: Graceful handling of network failures

Validates:

• Connection loss detection
• Automatic reconnection attempts
• Resume after reconnection
• User notification of connection issues

Failure Scenarios:

• Temporary network loss (WiFi disconnect)
• Server unavailability (503 responses)
• Timeout errors (slow connections)

17. Performance Tests

Tests: Upload performance benchmarks

Validates:

• 10MB file uploads in <30 seconds
• Memory efficiency for large files (50MB)
• No memory leaks during uploads
• CPU usage within acceptable limits

Performance Targets:

• Throughput: >2MB/s on localhost
• Memory: <100MB for 50MB file upload
• CPU: <50% average during upload

Test Page Requirements

Tests assume the following test page structure at https://localhost/livecomponents/test/upload:

Required HTML Elements

<div data-component-id="upload:test">
    <!-- File Input -->
    <input type="file" id="file-input" multiple />

    <!-- Upload Control Buttons -->
    <button id="upload-btn">Upload</button>
    <button id="resume-upload-btn" style="display:none">Resume Upload</button>
    <button id="cancel-upload-btn" style="display:none">Cancel Upload</button>

    <!-- Progress Display -->
    <div id="upload-progress">
        <div id="progress-bar">
            <div id="progress-fill" style="width: 0%"></div>
        </div>
        <span id="progress-text">0%</span>
        <span id="upload-speed">0 MB/s</span>
    </div>

    <!-- Status Display -->
    <div id="upload-status">Ready</div>

    <!-- File List (Fragment) -->
    <div data-lc-fragment="file-list">
        <!-- Uploaded files will be rendered here -->
    </div>

    <!-- Quarantine Status -->
    <div data-lc-fragment="quarantine-status">
        <!-- Virus scan status will be rendered here -->
    </div>
</div>

Required Component Actions

final readonly class UploadTestComponent extends LiveComponent
{
    #[Action]
    public function initializeUpload(string $filename, int $fileSize): array
    {
        // Return session ID and chunk configuration
        return [
            'sessionId' => Uuid::generate(),
            'chunkSize' => 512 * 1024, // 512KB
            'totalChunks' => ceil($fileSize / (512 * 1024))
        ];
    }

    #[Action]
    public function uploadChunk(
        string $sessionId,
        int $chunkIndex,
        string $chunkData,
        string $hash
    ): array {
        // Verify hash, store chunk, return progress
        return [
            'success' => true,
            'uploadedChunks' => $chunkIndex + 1,
            'progress' => (($chunkIndex + 1) / $totalChunks) * 100
        ];
    }

    #[Action]
    public function completeUpload(string $sessionId): array
    {
        // Assemble file, move to quarantine, trigger scan
        return [
            'success' => true,
            'fileId' => $fileId,
            'quarantineStatus' => 'scanning'
        ];
    }

    #[Action]
    public function cancelUpload(string $sessionId): array
    {
        // Clean up session and temporary chunks
        return ['success' => true];
    }
}

SSE Endpoint

#[Route(path: '/upload-progress/{sessionId}', method: Method::GET)]
public function uploadProgress(string $sessionId): SseStream
{
    return new SseStream(function() use ($sessionId) {
        while ($session = $this->getUploadSession($sessionId)) {
            yield [
                'type' => 'upload-progress',
                'sessionId' => $sessionId,
                'progress' => $session->getProgress(),
                'uploadedChunks' => $session->getUploadedChunks(),
                'totalChunks' => $session->getTotalChunks()
            ];

            sleep(1);

            if ($session->isComplete()) {
                break;
            }
        }
    });
}

Configuration

Environment Variables

# Chunked Upload Configuration
UPLOAD_CHUNK_SIZE=524288              # 512KB in bytes
UPLOAD_MAX_FILE_SIZE=104857600        # 100MB in bytes
UPLOAD_PARALLEL_CHUNKS=3              # Concurrent uploads
UPLOAD_RETRY_ATTEMPTS=3               # Max retries per chunk
UPLOAD_RETRY_BASE_DELAY=100           # Base delay in ms
UPLOAD_QUARANTINE_PATH=/var/quarantine
UPLOAD_VIRUS_SCAN_ENABLED=true

Test Configuration

// In chunked-upload.spec.js
const TEST_CONFIG = {
    baseUrl: 'https://localhost',
    testPagePath: '/livecomponents/test/upload',
    testFilesDir: './tests/tmp/upload-test-files',
    timeouts: {
        uploadSmall: 10000,    // 10s for small files
        uploadMedium: 30000,   // 30s for medium files
        uploadLarge: 120000    // 120s for large files
    }
};

Troubleshooting

Tests Timing Out

Symptoms:

• Upload tests exceed timeout limits
• Progress stuck at certain percentage

Solutions:

// Increase timeout for specific test
test('large file upload', async ({ page }) => {
    test.setTimeout(120000); // 120 seconds for large files
    // ...
});

File Generation Issues

Symptoms:

• Test files not created
• Permission errors in tmp directory

Solutions:

# Ensure tmp directory exists and is writable
mkdir -p tests/tmp/upload-test-files
chmod 755 tests/tmp/upload-test-files

# Clean up before running tests
rm -rf tests/tmp/upload-test-files/*

Network Request Monitoring Failures

Symptoms:

• Parallel upload detection fails
• Request interception not working

Solutions:

// Set up request monitoring before navigation
await page.route('**/live-component/**', route => {
    // Your intercept logic
    route.continue();
});

await page.goto('https://localhost/livecomponents/test/upload');

SSE Connection Issues

Symptoms:

• Progress updates not received
• SSE event listener not triggering

Solutions:

// Verify SSE connection
await page.evaluate(() => {
    const eventSource = new EventSource('/upload-progress/session-id');
    eventSource.onmessage = (event) => {
        console.log('SSE Event:', event.data);
    };
    eventSource.onerror = (error) => {
        console.error('SSE Error:', error);
    };
});

Memory Issues with Large Files

Symptoms:

• Browser crashes during large file uploads
• Out of memory errors

Solutions:

// Use smaller test files or increase browser memory
const browser = await chromium.launch({
    args: ['--max-old-space-size=4096'] // 4GB memory limit
});

// Or reduce test file sizes
const testFile = await createTestFile('test-medium.bin', 10); // 10MB instead of 50MB

Continuous Integration

GitHub Actions Integration

name: Chunked Upload E2E Tests

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  upload-tests:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - uses: actions/setup-node@v3
        with:
          node-version: '18'

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright
        run: npx playwright install --with-deps chromium

      - name: Start dev server
        run: make up

      - name: Run chunked upload tests
        run: npm run test:upload

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: upload-test-results
          path: test-results/

Best Practices

1. Test File Management

• Always clean up test files after tests
• Use unique filenames to avoid conflicts
• Store test files in tests/tmp/ directory
• Add tests/tmp/ to .gitignore

2. Network Simulation

• Test with realistic network conditions
• Simulate network failures and recovery
• Test on slow connections (throttling)
• Validate behavior during high latency

3. State Management

• Verify component state after each action
• Test state persistence across interactions
• Ensure proper cleanup on errors
• Validate state synchronization

4. Performance Testing

• Test with various file sizes (1MB, 10MB, 50MB)
• Monitor memory usage during uploads
• Measure throughput and latency
• Validate resource cleanup

5. Error Handling

• Test all failure scenarios
• Verify error messages are user-friendly
• Ensure proper cleanup on errors
• Test recovery mechanisms

Resources

• LiveComponents Upload Guide
• Playwright Testing Documentation
• Web Crypto API - SubtleCrypto
• Server-Sent Events

Support

For issues or questions:

1. Review this documentation
2. Check test output for specific errors
3. Consult LiveComponents upload guide
4. Review Playwright documentation for browser automation issues
5. Create GitHub issue with test output and error logs