linus/fahnen_esp32
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
fahnen_esp32/.pio/libdeps/esp01_1m/FastLED/VIDEO.md

13 KiB
Raw Permalink Blame History

Video Recording Feature for FastLED Web App

Implementation Status: ✅ COMPLETE - FULLY IMPLEMENTED

Last Updated: September 18, 2025 Commit: Refactored video recording code to fix bugs and improve reliability

Overview

A video recording feature has been implemented for the FastLED web application in src/platforms/wasm/compiler/. This feature allows users to capture both the canvas animations and audio output to create high-quality video recordings.

Features Implemented

1. Recording Button UI

  • Location: Upper right corner of the canvas
  • Visual States:
    • Idle: Gray circular button with a record icon (⚫)
    • Recording: Red pulsing button with a stop icon (⬛)
  • Animation: Smooth pulse effect when recording
  • Note: Stdout button should be moved to upper left of canvas to avoid conflict with record button placement

2. Video Capture Specifications

  • Default Format: MP4 with H.264 video codec and AAC audio codec
  • Fallback Formats: WebM with VP9/VP8 video codec and Opus audio codec (auto-detected based on browser support)
  • Video Bitrate: 10 Mbps (high quality)
  • Audio Bitrate: 128 kbps
  • Frame Rate: 30 FPS default (configurable to 60 FPS)
  • Resolution: Captures at canvas native resolution
  • Frame Logging: Each frame insertion is logged to console (e.g., "frame 16")
  • Automatic File Extension: Correct file extension (.mp4 or .webm) based on selected codec
  • Recording Statistics: Logs frame count, duration, FPS, and file size upon completion
  • Improved Error Handling: Graceful fallback when MediaRecorder is unavailable

3. Audio Capture

  • Source: Web Audio API AudioContext
  • Integration: Automatically captures audio if AudioContext is available
  • MP3 Player Integration:
    • Captures audio from the existing MP3 player module
    • Audio streams from MP3 playback are routed through the AudioContext
    • Mixed audio (synthesized + MP3) is captured in the recording
  • Audio Mixing: Combines all audio sources (synthesized sounds + MP3 playback) into a single stream
  • Fallback: Records video-only if audio is unavailable
  • Sync: Audio and video are automatically synchronized

4. User Interaction

  • Start Recording: Click the record button or press Ctrl+R / Cmd+R
  • Stop Recording: Click the button again (now showing stop icon)
  • File Save: Automatic download prompt when recording stops
  • Filename Format: fastled-recording-YYYY-MM-DD-HH-MM-SS.{mp4|webm} (extension matches codec)
  • Graceful Degradation: Record button automatically hidden if MediaRecorder unsupported

Frame Handling & Encoder Optimizations

Dropped Frame Detection & Recovery

The video recording system implements comprehensive dropped frame handling:

Detection Methods

  • Frame Counter Monitoring: Tracks expected vs. actual frame sequence
  • Timestamp Analysis: Compares MediaRecorder timestamps with canvas refresh rate
  • Performance Observer: Monitors frame presentation delays and skips

Recovery Strategies

// Dropped frame detection
let expectedFrameCount = 0;
let actualFrameCount = 0;
const frameDropThreshold = 2; // Allow 2 consecutive drops before intervention

mediaRecorder.addEventListener('dataavailable', (event) => {
  actualFrameCount++;
  const timestamp = performance.now();
  const expectedTimestamp = startTime + (expectedFrameCount * (1000 / fps));

  if (timestamp - expectedTimestamp > frameDropThreshold * (1000 / fps)) {
    console.warn(`Dropped frames detected: expected ${expectedFrameCount}, got ${actualFrameCount}`);
    // Trigger recovery actions
    handleDroppedFrames(expectedFrameCount - actualFrameCount);
  }

  console.log(`frame ${actualFrameCount}`);
  expectedFrameCount++;
});

function handleDroppedFrames(droppedCount) {
  // Option 1: Duplicate last frame to maintain timing
  // Option 2: Reduce capture frame rate temporarily
  // Option 3: Skip frame interpolation for smooth playback
}

Duplicate Frame Optimization

H.264 Encoder Optimizations

The H.264 encoder in MP4 format provides several optimizations for static or repetitive content:

  • P-frames (Predicted frames): Encode only differences from previous frames
  • B-frames (Bidirectional frames): Reference both previous and future frames for maximum compression
  • Temporal Compression: Automatically detects static regions and encodes efficiently
  • Rate Control: Adjusts bitrate dynamically based on content complexity

Implementation Details

// Configure encoder for optimal duplicate frame handling
const encoderConfig = {
  mimeType: 'video/mp4;codecs=h264,aac',
  videoBitsPerSecond: 10000000,
  audioBitsPerSecond: 128000,
  // H.264 specific optimizations
  keyFrameInterval: 30, // I-frame every 30 frames (1 second at 30fps)
  frameSkipping: true,  // Allow encoder to skip identical frames
  contentHint: 'motion' // Optimize for animation content
};

// Monitor encoding efficiency
mediaRecorder.addEventListener('start', () => {
  console.log('Encoder supports duplicate frame optimization: ',
    mediaRecorder.mimeType.includes('h264'));
});

Fallback Encoder Behavior

When MP4/H.264 is not available, the system falls back to WebM/VP9:

  • VP9 Optimization: Similar temporal compression but different algorithm
  • Opus Audio: Efficient silence detection and compression
  • WebM Container: Supports variable frame rates for dropped frame scenarios

Performance Monitoring

// Track encoding performance
const encodingStats = {
  framesEncoded: 0,
  framesDuplicated: 0,
  bytesGenerated: 0,
  compressionRatio: 0
};

mediaRecorder.addEventListener('dataavailable', (event) => {
  encodingStats.bytesGenerated += event.data.size;
  encodingStats.framesEncoded++;

  // Estimate duplicate frame optimization
  const expectedSize = (canvas.width * canvas.height * 3) / 8; // RGB to bytes
  const actualSize = event.data.size;
  const efficiency = 1 - (actualSize / expectedSize);

  if (efficiency > 0.8) {
    encodingStats.framesDuplicated++;
    console.log(`High compression frame ${encodingStats.framesEncoded} (${efficiency.toFixed(2)} efficiency)`);
  }
});

Technical Implementation

Files Created/Modified

  1. modules/video_recorder.js (NEW)

    • Core VideoRecorder class
    • MediaRecorder API integration
    • Canvas stream capture
    • Audio context integration with MP3 player support
    • Mixed audio capture (MP3 + synthesized sounds)
    • File save functionality

  2. index.html (MODIFIED)

    • Added record button HTML structure
    • SVG icons for record/stop states

  3. index.css (MODIFIED)

    • Record button styling
    • Animation effects (pulse, hover)
    • State transitions

  4. index.js (MODIFIED)

    • Video recorder initialization
    • Button event handling
    • Keyboard shortcut support
    • Global debug functions

API Documentation

MediaRecorder API Usage

// Canvas capture
canvas.captureStream(fps) // Creates MediaStream from canvas

// Audio capture with MP3 mixing
audioContext.createMediaStreamDestination() // Creates audio destination node

// MP3 player audio routing
mp3Player.connect(audioDestination) // Route MP3 audio to recording destination
synthesizer.connect(audioDestination) // Route synthesized audio to destination

// Combined stream with mixed audio
new MediaStream([
  ...videoStream.getVideoTracks(),
  ...audioDestination.stream.getAudioTracks() // Contains mixed audio from all sources
])

// Recording with mixed audio
mediaRecorder = new MediaRecorder(stream, {
  mimeType: 'video/mp4;codecs=h264,aac',  // Default: MP4 with H.264 video and AAC audio
  videoBitsPerSecond: 10000000,
  audioBitsPerSecond: 128000  // Captures all mixed audio sources
})

// Frame logging during recording
mediaRecorder.addEventListener('dataavailable', (event) => {
  console.log(`frame ${++frameCounter}`);  // Logs each frame insertion
});

Key Design Decisions

  1. File Save Timing: Files are saved AFTER recording stops (not when it starts)

    • Ensures precise start/stop timing for A/V sync
    • User sees save dialog after recording completes
    • No filename prompt interrupts recording start

  2. High Bitrate Choice: 10 Mbps video bitrate

    • Ensures high quality capture
    • Suitable for 60 FPS content
    • Can be compressed later if needed

  3. MP4/H.264 Format: Primary codec choice

    • Universal browser support for high-quality video
    • Good compression efficiency with AAC audio
    • Supports both video and audio tracks
    • WebM/VP9 fallback for browsers with limited MP4 support

Usage Instructions

For Users

  1. Click the gray record button in the upper-right corner of the canvas
  2. Button turns red and pulses to indicate recording
  3. Perform your LED animation
  4. Click the red button to stop recording
  5. Browser will prompt to save the video file

For Developers

Global Functions (Console Access)

// Get recorder instance
window.getVideoRecorder()

// Start recording programmatically
window.startVideoRecording()

// Stop recording programmatically
window.stopVideoRecording()

Keyboard Shortcuts

  • Ctrl+R / Cmd+R: Toggle recording on/off

Browser Compatibility

  • ✅ Chrome/Chromium (Full support)
  • ✅ Firefox (Full support)
  • ✅ Edge (Full support)
  • ⚠️ Safari (Limited codec support, uses fallback)

Recent Bug Fixes & Improvements

September 18, 2025 Refactor

  • Fixed codec configuration: Changed default from VP9 to H.264/MP4 for better compatibility
  • Improved audio routing: Enhanced audio connection logic for more reliable audio capture
  • Dynamic file extensions: Automatically uses correct extension (.mp4 or .webm) based on codec
  • Enhanced error handling: Added graceful fallback when MediaRecorder API is unavailable
  • Better resource cleanup: Improved disposal method to prevent memory leaks
  • Frame logging implementation: Added proper frame counting and logging as documented
  • Recording statistics: Added comprehensive logging of recording metrics (duration, FPS, file size)
  • Canvas content detection: Improved blank canvas detection without interfering with actual content
  • UI positioning fix: Moved stdout label to upper left to avoid conflicts with record button

Bug Fixes Addressed

  1. Inconsistent codec defaults - Now prioritizes H.264/MP4 over VP9/WebM
  2. Audio connection failures - Improved audio routing with better error handling
  3. Wrong file extensions - Files now use correct extension matching the codec
  4. Missing frame logging - Frame counter now properly logs each frame
  5. Memory leaks - Enhanced cleanup in disposal and error scenarios
  6. Canvas interference - Removed test pattern drawing that could interfere with content
  7. MediaRecorder detection - Added proper feature detection and graceful degradation

Future Enhancements (Optional)

  • Settings menu for quality presets
  • Frame rate selector (30/60 FPS toggle)
  • Recording timer display
  • Pause/resume functionality
  • Custom filename input
  • Multiple format export options
  • Recording preview before save
  • Audio level meters for MP3 and synthesized audio
  • Separate audio track controls (mute individual sources)
  • Audio normalization for mixed sources

Troubleshooting

No Audio in Recording

  • Check if browser has microphone permissions
  • Verify AudioContext is initialized
  • Some browsers require user interaction to start AudioContext
  • Ensure MP3 player audio nodes are properly connected to the recording destination
  • Verify that audio mixing is working (check browser console for audio node errors)

Recording Button Not Appearing

  • Check browser console for errors
  • Verify canvas element exists with ID "myCanvas"
  • Ensure JavaScript modules are loading correctly

Low Quality Video

  • Browser may be using fallback codec
  • Check available disk space
  • Try different browser for VP9 support

Dropped Frames Issues

  • High CPU Usage: Reduce canvas complexity or frame rate
  • Memory Pressure: Monitor browser memory usage, restart if needed
  • Background Tabs: Ensure recording tab has focus for optimal performance
  • Hardware Acceleration: Enable GPU acceleration in browser settings

Encoder Optimization Not Working

  • H.264 Unavailable: Check browser codec support with MediaRecorder.isTypeSupported()
  • Poor Compression: Verify content has static regions for optimization
  • Frame Skipping Disabled: Some browsers may not support frameSkipping parameter
  • Content Hint Ignored: Fallback to default encoding if contentHint not supported

Testing Checklist

Basic Functionality

  • Record button appears in correct position
  • Button changes color when recording
  • Pulse animation works during recording
  • Canvas content is captured correctly
  • Audio is captured (when available)
  • File saves with correct timestamp
  • Keyboard shortcut works
  • Multiple start/stop cycles work correctly
  • Memory is properly released after recording

Frame Handling & Optimization

  • Frame logging appears in console during recording
  • Dropped frame detection triggers warnings when appropriate
  • Static content shows high compression efficiency
  • H.264 encoder optimization is active when supported
  • Fallback to WebM/VP9 works when MP4 unavailable
  • Performance monitoring tracks encoding statistics
  • Recovery strategies activate during frame drops
  • Variable frame rate handling works under load
Reference in New Issue View Git Blame Copy Permalink