michaelschiemer/tests/e2e/livecomponents/CHUNKED-UPLOAD-TESTS.md

# LiveComponents Chunked Upload E2E Tests

Comprehensive end-to-end testing suite for the LiveComponents chunked upload system.

## Overview

This test suite validates the complete chunked upload functionality including:
- Upload session initialization and management
- Chunk splitting and parallel uploads
- SHA-256 integrity verification
- Progress tracking via SSE
- Resume capability for interrupted uploads
- Retry logic with exponential backoff
- Quarantine system integration
- Component state synchronization

## Quick Start

### Prerequisites

```bash
# Ensure Playwright is installed
npm install

# Install browser binaries
npx playwright install chromium

# Ensure development server is running
make up
```

### Running Tests

```bash
# Run all chunked upload tests
npm run test:upload

# Run with visible browser (for debugging)
npm run test:upload:headed

# Run with debug mode
npm run test:upload:debug

# Run specific test
npx playwright test chunked-upload.spec.js --grep "should upload chunks in parallel"
```

## Test Scenarios

### 1. Upload Session Initialization

**Tests:** Session creation with UUID format validation

**Validates:**
- Unique session ID generation
- UUID format (RFC 4122)
- Session metadata storage
- Client-server session sync

**Expected Behavior:**
- Session ID: `^[a-f0-9-]{36}$` (UUID v4 format)
- Session created within 500ms
- Session accessible via `window.__uploadSession`

### 2. Chunk Splitting Verification

**Tests:** File split into correct number of chunks based on size

**Validates:**
- Chunk size calculation (default: 512KB)
- Total chunks = ceil(fileSize / chunkSize)
- Chunk boundaries alignment

**Example:**
- 2MB file → 4 chunks (2048KB / 512KB = 4)
- 1MB file → 2 chunks (1024KB / 512KB = 2)
- 600KB file → 2 chunks (ceil(600KB / 512KB) = 2)

### 3. Parallel Chunk Uploads

**Tests:** Chunks upload concurrently (default: 3 parallel)

**Validates:**
- Multiple chunks upload simultaneously
- Concurrent request detection (time window analysis)
- Upload parallelization efficiency

**Metrics:**
- Expected concurrency: 3 simultaneous uploads
- Time window: Requests within 100ms considered parallel

### 4. Progress Tracking Accuracy

**Tests:** Real-time progress updates from 0% to 100%

**Validates:**
- Monotonic progress increase
- Accurate percentage calculation
- No progress regression
- Final state: 100% completion

**Progress Formula:**
```javascript
progress = (uploadedChunks / totalChunks) * 100
```

### 5. SHA-256 Integrity Verification

**Tests:** Hash calculation and verification for each chunk

**Validates:**
- Client-side hash generation (SHA-256)
- Server-side hash verification
- Hash format: 64 hexadecimal characters
- Upload rejection on hash mismatch

**Hash Calculation:**
```javascript
const hash = await crypto.subtle.digest('SHA-256', chunkData);
const hashHex = Array.from(new Uint8Array(hash))
    .map(b => b.toString(16).padStart(2, '0'))
    .join('');
```

### 6. Upload Interruption and Resume

**Tests:** Resume capability after page reload or connection loss

**Validates:**
- Uploaded chunks persistence
- Resume from last successful chunk
- Skip already uploaded chunks
- Session restoration

**Resume Workflow:**
1. Upload starts, some chunks complete
2. Interruption occurs (reload/disconnect)
3. Session restored with uploaded chunks info
4. Upload resumes from next chunk
5. Completion without re-uploading existing chunks

### 7. Retry Logic with Exponential Backoff

**Tests:** Failed chunks retry with increasing delays

**Validates:**
- Automatic retry on failure
- Exponential backoff strategy
- Maximum retry attempts (default: 3)
- Success after retries

**Backoff Strategy:**
```javascript
delay = baseDelay * Math.pow(2, attemptNumber)
// Attempt 1: 100ms
// Attempt 2: 200ms
// Attempt 3: 400ms
```

### 8. Concurrent Multi-File Uploads

**Tests:** Multiple files upload simultaneously with queue management

**Validates:**
- Independent upload sessions per file
- Progress tracking per file
- Queue management (max concurrent files)
- No session conflicts

**Expected Behavior:**
- Each file: unique session ID
- Independent progress tracking
- Proper resource cleanup

### 9. Real-time SSE Progress Updates

**Tests:** Server-Sent Events for live progress updates

**Validates:**
- SSE connection establishment
- Progress events received
- Event format and data accuracy
- Connection persistence

**SSE Event Format:**
```javascript
{
    type: 'upload-progress',
    sessionId: 'uuid',
    progress: 0-100,
    uploadedChunks: number,
    totalChunks: number,
    bytesUploaded: number,
    totalBytes: number
}
```

### 10. Upload Cancellation

**Tests:** User-initiated upload cancellation

**Validates:**
- Cancel button functionality
- Pending chunk requests abortion
- Session cleanup
- UI state reset

**Cancellation Steps:**
1. Upload in progress
2. Cancel button clicked
3. All pending requests aborted
4. Session marked as cancelled
5. Temporary chunks deleted

### 11. File Size Validation

**Tests:** File size limits enforcement

**Validates:**
- Maximum file size check (configurable)
- Client-side pre-validation
- Server-side validation
- Clear error messaging

**Default Limits:**
- Max file size: 100MB (configurable)
- Validation occurs before session creation

### 12. File Type Validation

**Tests:** Allowed file types enforcement

**Validates:**
- MIME type checking
- File extension validation
- Blocked file types rejection
- Whitelist/blacklist support

**Example Allowed Types:**
```javascript
allowedTypes: [
    'image/jpeg', 'image/png', 'image/gif',
    'application/pdf', 'application/zip'
]
```

### 13. Uploaded File Display

**Tests:** File appears in component after successful upload

**Validates:**
- Fragment update with file list
- File metadata display (name, size, date)
- DOM update verification
- Component state consistency

**Expected DOM:**
```html
<div data-lc-fragment="file-list">
    <div class="file-item" data-file-id="uuid">
        <span class="file-name">example.pdf</span>
        <span class="file-size">2.4 MB</span>
        <span class="file-date">2025-01-19</span>
    </div>
</div>
```

### 14. Quarantine System Integration

**Tests:** Uploaded files move to quarantine for virus scanning

**Validates:**
- File moved to quarantine directory
- Virus scan initiated (ClamAV/VirusTotal)
- Quarantine status tracking
- Clean/infected file handling

**Quarantine Workflow:**
1. Upload completes successfully
2. File moved to quarantine directory
3. Virus scan job queued
4. Component shows "Scanning..." status
5. Result: Clean → Available, Infected → Deleted

### 15. Component State Updates

**Tests:** Component state synchronizes after upload

**Validates:**
- State updates via LiveComponents protocol
- Fragment rendering with new data
- State persistence across interactions
- No state corruption

### 16. Network Interruption Handling

**Tests:** Graceful handling of network failures

**Validates:**
- Connection loss detection
- Automatic reconnection attempts
- Resume after reconnection
- User notification of connection issues

**Failure Scenarios:**
- Temporary network loss (WiFi disconnect)
- Server unavailability (503 responses)
- Timeout errors (slow connections)

### 17. Performance Tests

**Tests:** Upload performance benchmarks

**Validates:**
- 10MB file uploads in <30 seconds
- Memory efficiency for large files (50MB)
- No memory leaks during uploads
- CPU usage within acceptable limits

**Performance Targets:**
- **Throughput**: >2MB/s on localhost
- **Memory**: <100MB for 50MB file upload
- **CPU**: <50% average during upload

## Test Page Requirements

Tests assume the following test page structure at `https://localhost/livecomponents/test/upload`:

### Required HTML Elements

```html
<div data-component-id="upload:test">
    <!-- File Input -->
    <input type="file" id="file-input" multiple />

    <!-- Upload Control Buttons -->
    <button id="upload-btn">Upload</button>
    <button id="resume-upload-btn" style="display:none">Resume Upload</button>
    <button id="cancel-upload-btn" style="display:none">Cancel Upload</button>

    <!-- Progress Display -->
    <div id="upload-progress">
        <div id="progress-bar">
            <div id="progress-fill" style="width: 0%"></div>
        </div>
        <span id="progress-text">0%</span>
        <span id="upload-speed">0 MB/s</span>
    </div>

    <!-- Status Display -->
    <div id="upload-status">Ready</div>

    <!-- File List (Fragment) -->
    <div data-lc-fragment="file-list">
        <!-- Uploaded files will be rendered here -->
    </div>

    <!-- Quarantine Status -->
    <div data-lc-fragment="quarantine-status">
        <!-- Virus scan status will be rendered here -->
    </div>
</div>
```

### Required Component Actions

```php
final readonly class UploadTestComponent extends LiveComponent
{
    #[Action]
    public function initializeUpload(string $filename, int $fileSize): array
    {
        // Return session ID and chunk configuration
        return [
            'sessionId' => Uuid::generate(),
            'chunkSize' => 512 * 1024, // 512KB
            'totalChunks' => ceil($fileSize / (512 * 1024))
        ];
    }

    #[Action]
    public function uploadChunk(
        string $sessionId,
        int $chunkIndex,
        string $chunkData,
        string $hash
    ): array {
        // Verify hash, store chunk, return progress
        return [
            'success' => true,
            'uploadedChunks' => $chunkIndex + 1,
            'progress' => (($chunkIndex + 1) / $totalChunks) * 100
        ];
    }

    #[Action]
    public function completeUpload(string $sessionId): array
    {
        // Assemble file, move to quarantine, trigger scan
        return [
            'success' => true,
            'fileId' => $fileId,
            'quarantineStatus' => 'scanning'
        ];
    }

    #[Action]
    public function cancelUpload(string $sessionId): array
    {
        // Clean up session and temporary chunks
        return ['success' => true];
    }
}
```

### SSE Endpoint

```php
#[Route(path: '/upload-progress/{sessionId}', method: Method::GET)]
public function uploadProgress(string $sessionId): SseStream
{
    return new SseStream(function() use ($sessionId) {
        while ($session = $this->getUploadSession($sessionId)) {
            yield [
                'type' => 'upload-progress',
                'sessionId' => $sessionId,
                'progress' => $session->getProgress(),
                'uploadedChunks' => $session->getUploadedChunks(),
                'totalChunks' => $session->getTotalChunks()
            ];

            sleep(1);

            if ($session->isComplete()) {
                break;
            }
        }
    });
}
```

## Configuration

### Environment Variables

```env
# Chunked Upload Configuration
UPLOAD_CHUNK_SIZE=524288              # 512KB in bytes
UPLOAD_MAX_FILE_SIZE=104857600        # 100MB in bytes
UPLOAD_PARALLEL_CHUNKS=3              # Concurrent uploads
UPLOAD_RETRY_ATTEMPTS=3               # Max retries per chunk
UPLOAD_RETRY_BASE_DELAY=100           # Base delay in ms
UPLOAD_QUARANTINE_PATH=/var/quarantine
UPLOAD_VIRUS_SCAN_ENABLED=true
```

### Test Configuration

```javascript
// In chunked-upload.spec.js
const TEST_CONFIG = {
    baseUrl: 'https://localhost',
    testPagePath: '/livecomponents/test/upload',
    testFilesDir: './tests/tmp/upload-test-files',
    timeouts: {
        uploadSmall: 10000,    // 10s for small files
        uploadMedium: 30000,   // 30s for medium files
        uploadLarge: 120000    // 120s for large files
    }
};
```

## Troubleshooting

### Tests Timing Out

**Symptoms:**
- Upload tests exceed timeout limits
- Progress stuck at certain percentage

**Solutions:**
```javascript
// Increase timeout for specific test
test('large file upload', async ({ page }) => {
    test.setTimeout(120000); // 120 seconds for large files
    // ...
});
```

### File Generation Issues

**Symptoms:**
- Test files not created
- Permission errors in tmp directory

**Solutions:**
```bash
# Ensure tmp directory exists and is writable
mkdir -p tests/tmp/upload-test-files
chmod 755 tests/tmp/upload-test-files

# Clean up before running tests
rm -rf tests/tmp/upload-test-files/*
```

### Network Request Monitoring Failures

**Symptoms:**
- Parallel upload detection fails
- Request interception not working

**Solutions:**
```javascript
// Set up request monitoring before navigation
await page.route('**/live-component/**', route => {
    // Your intercept logic
    route.continue();
});

await page.goto('https://localhost/livecomponents/test/upload');
```

### SSE Connection Issues

**Symptoms:**
- Progress updates not received
- SSE event listener not triggering

**Solutions:**
```javascript
// Verify SSE connection
await page.evaluate(() => {
    const eventSource = new EventSource('/upload-progress/session-id');
    eventSource.onmessage = (event) => {
        console.log('SSE Event:', event.data);
    };
    eventSource.onerror = (error) => {
        console.error('SSE Error:', error);
    };
});
```

### Memory Issues with Large Files

**Symptoms:**
- Browser crashes during large file uploads
- Out of memory errors

**Solutions:**
```javascript
// Use smaller test files or increase browser memory
const browser = await chromium.launch({
    args: ['--max-old-space-size=4096'] // 4GB memory limit
});

// Or reduce test file sizes
const testFile = await createTestFile('test-medium.bin', 10); // 10MB instead of 50MB
```

## Continuous Integration

### GitHub Actions Integration

```yaml
name: Chunked Upload E2E Tests

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  upload-tests:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - uses: actions/setup-node@v3
        with:
          node-version: '18'

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright
        run: npx playwright install --with-deps chromium

      - name: Start dev server
        run: make up

      - name: Run chunked upload tests
        run: npm run test:upload

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: upload-test-results
          path: test-results/
```

## Best Practices

### 1. Test File Management
- Always clean up test files after tests
- Use unique filenames to avoid conflicts
- Store test files in `tests/tmp/` directory
- Add `tests/tmp/` to `.gitignore`

### 2. Network Simulation
- Test with realistic network conditions
- Simulate network failures and recovery
- Test on slow connections (throttling)
- Validate behavior during high latency

### 3. State Management
- Verify component state after each action
- Test state persistence across interactions
- Ensure proper cleanup on errors
- Validate state synchronization

### 4. Performance Testing
- Test with various file sizes (1MB, 10MB, 50MB)
- Monitor memory usage during uploads
- Measure throughput and latency
- Validate resource cleanup

### 5. Error Handling
- Test all failure scenarios
- Verify error messages are user-friendly
- Ensure proper cleanup on errors
- Test recovery mechanisms

## Resources

- [LiveComponents Upload Guide](../../../src/Framework/LiveComponents/docs/UPLOAD-GUIDE.md)
- [Playwright Testing Documentation](https://playwright.dev/docs/intro)
- [Web Crypto API - SubtleCrypto](https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto)
- [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events)

## Support

For issues or questions:
1. Review this documentation
2. Check test output for specific errors
3. Consult LiveComponents upload guide
4. Review Playwright documentation for browser automation issues
5. Create GitHub issue with test output and error logs