imported from "final" folder

2025-11-28 12:12:50 +01:00
parent f9288986cf
commit ff8e725b35
1061 changed files with 225150 additions and 96 deletions
--- a/.pio/libdeps/esp01_1m/FastLED/VIDEO.md
+++ b/.pio/libdeps/esp01_1m/FastLED/VIDEO.md
@@ -0,0 +1,352 @@
+# Video Recording Feature for FastLED Web App
+
+## Implementation Status: ✅ COMPLETE - FULLY IMPLEMENTED
+
+**Last Updated**: September 18, 2025
+**Commit**: Refactored video recording code to fix bugs and improve reliability
+
+### Overview
+A video recording feature has been implemented for the FastLED web application in `src/platforms/wasm/compiler/`. This feature allows users to capture both the canvas animations and audio output to create high-quality video recordings.
+
+## Features Implemented
+
+### 1. Recording Button UI
+- **Location**: Upper right corner of the canvas
+- **Visual States**:
+  - **Idle**: Gray circular button with a record icon (⚫)
+  - **Recording**: Red pulsing button with a stop icon (⬛)
+- **Animation**: Smooth pulse effect when recording
+- **Note**: Stdout button should be moved to upper left of canvas to avoid conflict with record button placement
+
+### 2. Video Capture Specifications
+- **Default Format**: MP4 with H.264 video codec and AAC audio codec
+- **Fallback Formats**: WebM with VP9/VP8 video codec and Opus audio codec (auto-detected based on browser support)
+- **Video Bitrate**: 10 Mbps (high quality)
+- **Audio Bitrate**: 128 kbps
+- **Frame Rate**: 30 FPS default (configurable to 60 FPS)
+- **Resolution**: Captures at canvas native resolution
+- **Frame Logging**: Each frame insertion is logged to console (e.g., "frame 16")
+- **Automatic File Extension**: Correct file extension (.mp4 or .webm) based on selected codec
+- **Recording Statistics**: Logs frame count, duration, FPS, and file size upon completion
+- **Improved Error Handling**: Graceful fallback when MediaRecorder is unavailable
+
+### 3. Audio Capture
+- **Source**: Web Audio API AudioContext
+- **Integration**: Automatically captures audio if AudioContext is available
+- **MP3 Player Integration**:
+  - Captures audio from the existing MP3 player module
+  - Audio streams from MP3 playback are routed through the AudioContext
+  - Mixed audio (synthesized + MP3) is captured in the recording
+- **Audio Mixing**: Combines all audio sources (synthesized sounds + MP3 playback) into a single stream
+- **Fallback**: Records video-only if audio is unavailable
+- **Sync**: Audio and video are automatically synchronized
+
+### 4. User Interaction
+- **Start Recording**: Click the record button or press `Ctrl+R` / `Cmd+R`
+- **Stop Recording**: Click the button again (now showing stop icon)
+- **File Save**: Automatic download prompt when recording stops
+- **Filename Format**: `fastled-recording-YYYY-MM-DD-HH-MM-SS.{mp4|webm}` (extension matches codec)
+- **Graceful Degradation**: Record button automatically hidden if MediaRecorder unsupported
+
+## Frame Handling & Encoder Optimizations
+
+### Dropped Frame Detection & Recovery
+The video recording system implements comprehensive dropped frame handling:
+
+#### Detection Methods
+- **Frame Counter Monitoring**: Tracks expected vs. actual frame sequence
+- **Timestamp Analysis**: Compares MediaRecorder timestamps with canvas refresh rate
+- **Performance Observer**: Monitors frame presentation delays and skips
+
+#### Recovery Strategies
+```javascript
+// Dropped frame detection
+let expectedFrameCount = 0;
+let actualFrameCount = 0;
+const frameDropThreshold = 2; // Allow 2 consecutive drops before intervention
+
+mediaRecorder.addEventListener('dataavailable', (event) => {
+  actualFrameCount++;
+  const timestamp = performance.now();
+  const expectedTimestamp = startTime + (expectedFrameCount * (1000 / fps));
+
+  if (timestamp - expectedTimestamp > frameDropThreshold * (1000 / fps)) {
+    console.warn(`Dropped frames detected: expected ${expectedFrameCount}, got ${actualFrameCount}`);
+    // Trigger recovery actions
+    handleDroppedFrames(expectedFrameCount - actualFrameCount);
+  }
+
+  console.log(`frame ${actualFrameCount}`);
+  expectedFrameCount++;
+});
+
+function handleDroppedFrames(droppedCount) {
+  // Option 1: Duplicate last frame to maintain timing
+  // Option 2: Reduce capture frame rate temporarily
+  // Option 3: Skip frame interpolation for smooth playback
+}
+```
+
+### Duplicate Frame Optimization
+
+#### H.264 Encoder Optimizations
+The H.264 encoder in MP4 format provides several optimizations for static or repetitive content:
+
+- **P-frames (Predicted frames)**: Encode only differences from previous frames
+- **B-frames (Bidirectional frames)**: Reference both previous and future frames for maximum compression
+- **Temporal Compression**: Automatically detects static regions and encodes efficiently
+- **Rate Control**: Adjusts bitrate dynamically based on content complexity
+
+#### Implementation Details
+```javascript
+// Configure encoder for optimal duplicate frame handling
+const encoderConfig = {
+  mimeType: 'video/mp4;codecs=h264,aac',
+  videoBitsPerSecond: 10000000,
+  audioBitsPerSecond: 128000,
+  // H.264 specific optimizations
+  keyFrameInterval: 30, // I-frame every 30 frames (1 second at 30fps)
+  frameSkipping: true,  // Allow encoder to skip identical frames
+  contentHint: 'motion' // Optimize for animation content
+};
+
+// Monitor encoding efficiency
+mediaRecorder.addEventListener('start', () => {
+  console.log('Encoder supports duplicate frame optimization: ',
+    mediaRecorder.mimeType.includes('h264'));
+});
+```
+
+#### Fallback Encoder Behavior
+When MP4/H.264 is not available, the system falls back to WebM/VP9:
+
+- **VP9 Optimization**: Similar temporal compression but different algorithm
+- **Opus Audio**: Efficient silence detection and compression
+- **WebM Container**: Supports variable frame rates for dropped frame scenarios
+
+#### Performance Monitoring
+```javascript
+// Track encoding performance
+const encodingStats = {
+  framesEncoded: 0,
+  framesDuplicated: 0,
+  bytesGenerated: 0,
+  compressionRatio: 0
+};
+
+mediaRecorder.addEventListener('dataavailable', (event) => {
+  encodingStats.bytesGenerated += event.data.size;
+  encodingStats.framesEncoded++;
+
+  // Estimate duplicate frame optimization
+  const expectedSize = (canvas.width * canvas.height * 3) / 8; // RGB to bytes
+  const actualSize = event.data.size;
+  const efficiency = 1 - (actualSize / expectedSize);
+
+  if (efficiency > 0.8) {
+    encodingStats.framesDuplicated++;
+    console.log(`High compression frame ${encodingStats.framesEncoded} (${efficiency.toFixed(2)} efficiency)`);
+  }
+});
+```
+
+## Technical Implementation
+
+### Files Created/Modified
+
+1. **`modules/video_recorder.js`** (NEW)
+   - Core VideoRecorder class
+   - MediaRecorder API integration
+   - Canvas stream capture
+   - Audio context integration with MP3 player support
+   - Mixed audio capture (MP3 + synthesized sounds)
+   - File save functionality
+
+2. **`index.html`** (MODIFIED)
+   - Added record button HTML structure
+   - SVG icons for record/stop states
+
+3. **`index.css`** (MODIFIED)
+   - Record button styling
+   - Animation effects (pulse, hover)
+   - State transitions
+
+4. **`index.js`** (MODIFIED)
+   - Video recorder initialization
+   - Button event handling
+   - Keyboard shortcut support
+   - Global debug functions
+
+## API Documentation
+
+### MediaRecorder API Usage
+
+```javascript
+// Canvas capture
+canvas.captureStream(fps) // Creates MediaStream from canvas
+
+// Audio capture with MP3 mixing
+audioContext.createMediaStreamDestination() // Creates audio destination node
+
+// MP3 player audio routing
+mp3Player.connect(audioDestination) // Route MP3 audio to recording destination
+synthesizer.connect(audioDestination) // Route synthesized audio to destination
+
+// Combined stream with mixed audio
+new MediaStream([
+  ...videoStream.getVideoTracks(),
+  ...audioDestination.stream.getAudioTracks() // Contains mixed audio from all sources
+])
+
+// Recording with mixed audio
+mediaRecorder = new MediaRecorder(stream, {
+  mimeType: 'video/mp4;codecs=h264,aac',  // Default: MP4 with H.264 video and AAC audio
+  videoBitsPerSecond: 10000000,
+  audioBitsPerSecond: 128000  // Captures all mixed audio sources
+})
+
+// Frame logging during recording
+mediaRecorder.addEventListener('dataavailable', (event) => {
+  console.log(`frame ${++frameCounter}`);  // Logs each frame insertion
+});
+```
+
+### Key Design Decisions
+
+1. **File Save Timing**: Files are saved AFTER recording stops (not when it starts)
+   - Ensures precise start/stop timing for A/V sync
+   - User sees save dialog after recording completes
+   - No filename prompt interrupts recording start
+
+2. **High Bitrate Choice**: 10 Mbps video bitrate
+   - Ensures high quality capture
+   - Suitable for 60 FPS content
+   - Can be compressed later if needed
+
+3. **MP4/H.264 Format**: Primary codec choice
+   - Universal browser support for high-quality video
+   - Good compression efficiency with AAC audio
+   - Supports both video and audio tracks
+   - WebM/VP9 fallback for browsers with limited MP4 support
+
+## Usage Instructions
+
+### For Users
+1. Click the gray record button in the upper-right corner of the canvas
+2. Button turns red and pulses to indicate recording
+3. Perform your LED animation
+4. Click the red button to stop recording
+5. Browser will prompt to save the video file
+
+### For Developers
+
+#### Global Functions (Console Access)
+```javascript
+// Get recorder instance
+window.getVideoRecorder()
+
+// Start recording programmatically
+window.startVideoRecording()
+
+// Stop recording programmatically
+window.stopVideoRecording()
+```
+
+#### Keyboard Shortcuts
+- `Ctrl+R` / `Cmd+R`: Toggle recording on/off
+
+## Browser Compatibility
+
+- ✅ Chrome/Chromium (Full support)
+- ✅ Firefox (Full support)
+- ✅ Edge (Full support)
+- ⚠️ Safari (Limited codec support, uses fallback)
+
+## Recent Bug Fixes & Improvements
+
+### September 18, 2025 Refactor
+- **Fixed codec configuration**: Changed default from VP9 to H.264/MP4 for better compatibility
+- **Improved audio routing**: Enhanced audio connection logic for more reliable audio capture
+- **Dynamic file extensions**: Automatically uses correct extension (.mp4 or .webm) based on codec
+- **Enhanced error handling**: Added graceful fallback when MediaRecorder API is unavailable
+- **Better resource cleanup**: Improved disposal method to prevent memory leaks
+- **Frame logging implementation**: Added proper frame counting and logging as documented
+- **Recording statistics**: Added comprehensive logging of recording metrics (duration, FPS, file size)
+- **Canvas content detection**: Improved blank canvas detection without interfering with actual content
+- **UI positioning fix**: Moved stdout label to upper left to avoid conflicts with record button
+
+### Bug Fixes Addressed
+1. **Inconsistent codec defaults** - Now prioritizes H.264/MP4 over VP9/WebM
+2. **Audio connection failures** - Improved audio routing with better error handling
+3. **Wrong file extensions** - Files now use correct extension matching the codec
+4. **Missing frame logging** - Frame counter now properly logs each frame
+5. **Memory leaks** - Enhanced cleanup in disposal and error scenarios
+6. **Canvas interference** - Removed test pattern drawing that could interfere with content
+7. **MediaRecorder detection** - Added proper feature detection and graceful degradation
+
+## Future Enhancements (Optional)
+
+- [ ] Settings menu for quality presets
+- [ ] Frame rate selector (30/60 FPS toggle)
+- [ ] Recording timer display
+- [ ] Pause/resume functionality
+- [ ] Custom filename input
+- [ ] Multiple format export options
+- [ ] Recording preview before save
+- [ ] Audio level meters for MP3 and synthesized audio
+- [ ] Separate audio track controls (mute individual sources)
+- [ ] Audio normalization for mixed sources
+
+## Troubleshooting
+
+### No Audio in Recording
+- Check if browser has microphone permissions
+- Verify AudioContext is initialized
+- Some browsers require user interaction to start AudioContext
+- Ensure MP3 player audio nodes are properly connected to the recording destination
+- Verify that audio mixing is working (check browser console for audio node errors)
+
+### Recording Button Not Appearing
+- Check browser console for errors
+- Verify canvas element exists with ID "myCanvas"
+- Ensure JavaScript modules are loading correctly
+
+### Low Quality Video
+- Browser may be using fallback codec
+- Check available disk space
+- Try different browser for VP9 support
+
+### Dropped Frames Issues
+- **High CPU Usage**: Reduce canvas complexity or frame rate
+- **Memory Pressure**: Monitor browser memory usage, restart if needed
+- **Background Tabs**: Ensure recording tab has focus for optimal performance
+- **Hardware Acceleration**: Enable GPU acceleration in browser settings
+
+### Encoder Optimization Not Working
+- **H.264 Unavailable**: Check browser codec support with `MediaRecorder.isTypeSupported()`
+- **Poor Compression**: Verify content has static regions for optimization
+- **Frame Skipping Disabled**: Some browsers may not support frameSkipping parameter
+- **Content Hint Ignored**: Fallback to default encoding if contentHint not supported
+
+## Testing Checklist
+
+### Basic Functionality
+- [x] Record button appears in correct position
+- [x] Button changes color when recording
+- [x] Pulse animation works during recording
+- [x] Canvas content is captured correctly
+- [x] Audio is captured (when available)
+- [x] File saves with correct timestamp
+- [x] Keyboard shortcut works
+- [x] Multiple start/stop cycles work correctly
+- [x] Memory is properly released after recording
+
+### Frame Handling & Optimization
+- [x] Frame logging appears in console during recording
+- [x] Dropped frame detection triggers warnings when appropriate
+- [x] Static content shows high compression efficiency
+- [x] H.264 encoder optimization is active when supported
+- [x] Fallback to WebM/VP9 works when MP4 unavailable
+- [x] Performance monitoring tracks encoding statistics
+- [x] Recovery strategies activate during frame drops
+- [x] Variable frame rate handling works under load