# RMT5 Worker Pool Implementation ## Problem Statement ESP32 RMT5 has hardware limitations on the number of LED strips it can control simultaneously: - **ESP32**: 8 RMT channels maximum - **ESP32-S2/S3**: 4 RMT channels maximum - **ESP32-C3/H2**: 2 RMT channels maximum Currently, FastLED's RMT5 implementation creates a one-to-one mapping between `RmtController5` instances and RMT hardware channels. This means you can only control K strips simultaneously, where K is the hardware limit. **Goal**: Implement a worker pool system to support N strips where N > K, allowing any reasonable number of LED strips to be controlled by recycling RMT workers. ## Current RMT5 Architecture ### Class Hierarchy ```cpp ClocklessController └── RmtController5 (mRMTController) └── IRmtStrip* (mLedStrip) └── RmtStrip (concrete implementation) └── led_strip_handle_t (mStrip) // ESP-IDF handle └── rmt_channel_handle_t // Hardware channel ``` ### Current Flow 1. `RmtController5::loadPixelData()` creates `IRmtStrip` on first call 2. `IRmtStrip::Create()` calls `led_strip_new_rmt_device()` which allocates a hardware RMT channel 3. Channel remains allocated until `RmtStrip` destructor calls `led_strip_del()` 4. No sharing or pooling exists ## RMT4 Worker Pool Reference RMT4 implements a sophisticated worker pool system: ### Key Components - **`gControllers[FASTLED_RMT_MAX_CONTROLLERS]`**: Array of all registered controllers - **`gOnChannel[FASTLED_RMT_MAX_CHANNELS]`**: Currently active controllers per channel - **Global counters**: `gNumStarted`, `gNumDone`, `gNext` for coordination - **Semaphore coordination**: `gTX_sem` for synchronization ### RMT4 Worker Pool Flow 1. All controllers register in `gControllers[]` during construction 2. `showPixels()` triggers batch processing when `gNumStarted == gNumControllers` 3. First K controllers start immediately on available channels 4. Remaining controllers queue until channels become available 5. `doneOnChannel()` callback releases channels and starts next queued controller 6. Process continues until all controllers complete ## Proposed RMT5 Worker Pool Architecture ### Core Design Principles 1. **Backward Compatibility**: Existing `RmtController5` API remains unchanged 2. **Transparent Pooling**: Controllers don't know they're sharing workers 3. **Automatic Resource Management**: Workers handle setup/teardown automatically 4. **Thread Safety**: Pool operations are atomic and interrupt-safe ### New Components #### 1. RmtWorkerPool (Singleton) ```cpp class RmtWorkerPool { public: static RmtWorkerPool& getInstance(); // Worker management RmtWorker* acquireWorker(const RmtWorkerConfig& config); void releaseWorker(RmtWorker* worker); // Coordination void registerController(RmtController5* controller); void unregisterController(RmtController5* controller); void executeDrawCycle(); private: fl::vector mAvailableWorkers; fl::vector mBusyWorkers; fl::vector mRegisteredControllers; // Synchronization SemaphoreHandle_t mPoolMutex; SemaphoreHandle_t mDrawSemaphore; // State tracking int mActiveDrawCount; int mCompletedDrawCount; }; ``` #### 2. RmtWorker (Replaceable RMT Resource) ```cpp class RmtWorker { public: // Configuration for worker setup struct Config { int pin; uint32_t ledCount; bool isRgbw; uint32_t t0h, t0l, t1h, t1l, reset; IRmtStrip::DmaMode dmaMode; }; // Worker lifecycle bool configure(const Config& config); void loadPixelData(fl::PixelIterator& pixels); void startTransmission(); void waitForCompletion(); // State management bool isAvailable() const; bool isConfiguredFor(const Config& config) const; void reset(); // Callbacks void onTransmissionComplete(); private: IRmtStrip* mCurrentStrip; Config mCurrentConfig; bool mIsAvailable; bool mTransmissionActive; RmtWorkerPool* mPool; // Back reference for release }; ``` #### 3. Modified RmtController5 ```cpp class RmtController5 { public: // Existing API unchanged RmtController5(int DATA_PIN, int T1, int T2, int T3, DmaMode dma_mode); void loadPixelData(PixelIterator &pixels); void showPixels(); private: // New pooled implementation RmtWorkerConfig mWorkerConfig; fl::vector mPixelBuffer; // Persistent buffer bool mRegisteredWithPool; // Remove direct IRmtStrip ownership // IRmtStrip *mLedStrip = nullptr; // REMOVED }; ``` ### Worker Pool Operation Flow #### Registration Phase (Constructor) ```cpp RmtController5::RmtController5(int DATA_PIN, int T1, int T2, int T3, DmaMode dma_mode) : mPin(DATA_PIN), mT1(T1), mT2(T2), mT3(T3), mDmaMode(dma_mode) { // Configure worker requirements mWorkerConfig = {DATA_PIN, 0, false, t0h, t0l, t1h, t1l, 280, convertDmaMode(dma_mode)}; // Register with pool RmtWorkerPool::getInstance().registerController(this); mRegisteredWithPool = true; } ``` #### Data Loading Phase ```cpp void RmtController5::loadPixelData(PixelIterator &pixels) { // Update worker config with actual pixel count mWorkerConfig.ledCount = pixels.size(); mWorkerConfig.isRgbw = pixels.get_rgbw().active(); // Store pixel data in persistent buffer storePixelData(pixels); } void RmtController5::storePixelData(PixelIterator &pixels) { const int bytesPerPixel = mWorkerConfig.isRgbw ? 4 : 3; const int bufferSize = mWorkerConfig.ledCount * bytesPerPixel; mPixelBuffer.resize(bufferSize); // Copy pixel data to persistent buffer uint8_t* bufPtr = mPixelBuffer.data(); if (mWorkerConfig.isRgbw) { while (pixels.has(1)) { uint8_t r, g, b, w; pixels.loadAndScaleRGBW(&r, &g, &b, &w); *bufPtr++ = r; *bufPtr++ = g; *bufPtr++ = b; *bufPtr++ = w; pixels.advanceData(); pixels.stepDithering(); } } else { while (pixels.has(1)) { uint8_t r, g, b; pixels.loadAndScaleRGB(&r, &g, &b); *bufPtr++ = r; *bufPtr++ = g; *bufPtr++ = b; pixels.advanceData(); pixels.stepDithering(); } } } ``` #### Execution Phase (Coordinated Draw) ```cpp void RmtController5::showPixels() { // Trigger coordinated draw cycle RmtWorkerPool::getInstance().executeDrawCycle(); } void RmtWorkerPool::executeDrawCycle() { // Similar to RMT4 coordination logic mActiveDrawCount = 0; mCompletedDrawCount = 0; // Take draw semaphore xSemaphoreTake(mDrawSemaphore, portMAX_DELAY); // Start as many controllers as we have workers int startedCount = 0; for (auto* controller : mRegisteredControllers) { if (startedCount < mAvailableWorkers.size()) { startController(controller); startedCount++; } } // Wait for all controllers to complete while (mCompletedDrawCount < mRegisteredControllers.size()) { xSemaphoreTake(mDrawSemaphore, portMAX_DELAY); // Start next queued controller if workers available startNextQueuedController(); xSemaphoreGive(mDrawSemaphore); } } void RmtWorkerPool::startController(RmtController5* controller) { // Acquire worker from pool RmtWorker* worker = acquireWorker(controller->getWorkerConfig()); if (!worker) { // This should not happen if pool is sized correctly return; } // Configure worker for this controller worker->configure(controller->getWorkerConfig()); // Load pixel data from controller's persistent buffer loadPixelDataToWorker(worker, controller); // Start transmission worker->startTransmission(); mActiveDrawCount++; } void RmtWorkerPool::onWorkerComplete(RmtWorker* worker) { // Called from worker's completion callback mCompletedDrawCount++; // Release worker back to pool releaseWorker(worker); // Signal main thread xSemaphoreGive(mDrawSemaphore); } ``` ### Worker Reconfiguration Strategy #### Efficient Worker Reuse ```cpp bool RmtWorker::configure(const Config& newConfig) { // Check if reconfiguration is needed if (isConfiguredFor(newConfig)) { return true; // Already configured correctly } // Tear down current configuration if (mCurrentStrip) { // Wait for any pending transmission if (mTransmissionActive) { mCurrentStrip->waitDone(); } // Clean shutdown delete mCurrentStrip; mCurrentStrip = nullptr; } // Create new strip with new configuration mCurrentStrip = IRmtStrip::Create( newConfig.pin, newConfig.ledCount, newConfig.isRgbw, newConfig.t0h, newConfig.t0l, newConfig.t1h, newConfig.t1l, newConfig.reset, newConfig.dmaMode ); if (!mCurrentStrip) { return false; } mCurrentConfig = newConfig; return true; } ``` #### Pin State Management ```cpp void RmtWorker::reset() { if (mCurrentStrip) { // Ensure transmission is complete if (mTransmissionActive) { mCurrentStrip->waitDone(); } // Set pin to safe state before teardown gpio_set_level((gpio_num_t)mCurrentConfig.pin, 0); gpio_set_direction((gpio_num_t)mCurrentConfig.pin, GPIO_MODE_OUTPUT); // Clean up strip delete mCurrentStrip; mCurrentStrip = nullptr; } mTransmissionActive = false; mIsAvailable = true; } ``` ### Memory Management Strategy #### Persistent Pixel Buffers Each `RmtController5` maintains its own pixel buffer to avoid data races: ```cpp class RmtController5 { private: fl::vector mPixelBuffer; // Persistent storage RmtWorkerConfig mWorkerConfig; // Configuration cache void storePixelData(PixelIterator& pixels); void restorePixelData(RmtWorker* worker); }; ``` #### Worker Pool Sizing ```cpp void RmtWorkerPool::initialize() { // Determine hardware channel count int maxChannels = getHardwareChannelCount(); // Create one worker per hardware channel mAvailableWorkers.reserve(maxChannels); for (int i = 0; i < maxChannels; i++) { mAvailableWorkers.push_back(new RmtWorker()); } } int RmtWorkerPool::getHardwareChannelCount() { #if CONFIG_IDF_TARGET_ESP32 return 8; #elif CONFIG_IDF_TARGET_ESP32S2 || CONFIG_IDF_TARGET_ESP32S3 return 4; #elif CONFIG_IDF_TARGET_ESP32C3 || CONFIG_IDF_TARGET_ESP32H2 return 2; #else return 2; // Conservative default #endif } ``` ## Key Implementation Insights ### Critical Async Decision Point The worker pool must make a **per-controller decision** at `showPixels()` time: ```cpp void RmtController5::showPixels() { RmtWorkerPool& pool = RmtWorkerPool::getInstance(); // CRITICAL: This decision determines async vs blocking behavior if (pool.hasAvailableWorker()) { // ASYNC PATH: Start immediately and return pool.startControllerImmediate(this); return; // Returns immediately - preserves async! } else { // BLOCKING PATH: This specific controller must wait pool.startControllerQueued(this); // May block with polling } } ``` ### ESP-IDF RMT Channel Management Direct integration with ESP-IDF RMT5 APIs instead of using led_strip wrapper: ```cpp class RmtWorker { private: rmt_channel_handle_t mChannel; rmt_encoder_handle_t mEncoder; public: // Direct RMT channel creation for maximum control bool createChannel(int pin) { rmt_tx_channel_config_t config = { .gpio_num = pin, .clk_src = RMT_CLK_SRC_DEFAULT, .resolution_hz = 10000000, .mem_block_symbols = 64, .trans_queue_depth = 1, // Single transmission per worker }; ESP_ERROR_CHECK(rmt_new_tx_channel(&config, &mChannel)); // Register callback for async completion rmt_tx_event_callbacks_t callbacks = { .on_trans_done = onTransComplete, }; ESP_ERROR_CHECK(rmt_tx_register_event_callbacks(mChannel, &callbacks, this)); return true; } void transmitAsync(uint8_t* pixelData, size_t dataSize) { // Direct transmission - bypasses led_strip wrapper ESP_ERROR_CHECK(rmt_enable(mChannel)); ESP_ERROR_CHECK(rmt_transmit(mChannel, mEncoder, pixelData, dataSize, &mTxConfig)); // Returns immediately - async transmission started } }; ``` ### Polling Strategy Implementation Use `delayMicroseconds(100)` only for queued controllers: ```cpp void RmtWorkerPool::startControllerQueued(RmtController5* controller) { // Add to queue mQueuedControllers.push_back(controller); // Poll until this controller gets a worker while (true) { if (RmtWorker* worker = tryAcquireWorker()) { // Remove from queue and start mQueuedControllers.remove(controller); startControllerImmediate(controller, worker); break; } // Brief delay to prevent busy-wait delayMicroseconds(100); // Yield periodically for FreeRTOS static uint32_t pollCount = 0; if (++pollCount % 50 == 0) { yield(); } } } ``` ## Implementation Plan ### Phase 1: Core Infrastructure 1. **Create `RmtWorkerPool` singleton** - Basic worker management (acquire/release) - Controller registration system - Thread-safe operations with mutexes 2. **Implement `RmtWorker` class** - Worker lifecycle management - Configuration and reconfiguration logic - Completion callbacks 3. **Modify `RmtController5`** - Add persistent pixel buffer - Integrate with worker pool - Maintain backward-compatible API ### Phase 2: Coordination Logic 1. **Implement coordinated draw cycle** - Batch processing similar to RMT4 - Semaphore-based synchronization - Queue management for excess controllers 2. **Add worker completion handling** - Async completion callbacks - Automatic worker recycling - Next controller startup ### Phase 3: Optimization & Safety 1. **Optimize worker reconfiguration** - Minimize teardown/setup when possible - Cache compatible configurations - Efficient pin state management 2. **Add error handling** - Worker allocation failures - Transmission errors - Recovery mechanisms 3. **Memory optimization** - Minimize buffer copying - Efficient pixel data transfer - Memory pool for workers ### Phase 4: Testing & Integration 1. **Unit tests** - Worker pool operations - Controller coordination - Error scenarios 2. **Integration testing** - Multiple strip configurations - High load scenarios - Hardware limit validation 3. **Performance benchmarking** - Throughput comparison with RMT4 - Memory usage analysis - Latency measurements ## CRITICAL: Async Behavior Preservation **Key Requirement**: RMT5 currently provides async drawing where `endShowLeds()` returns immediately without waiting. This must be preserved when N ≤ K, and only use polling/waiting when N > K. ### Current RMT5 Async Flow ```cpp // Current behavior - MUST PRESERVE when N ≤ K void ClocklessController::endShowLeds(void *data) { CPixelLEDController::endShowLeds(data); mRMTController.showPixels(); // Calls drawAsync() - returns immediately! } ``` ### Async Strategy for Worker Pool #### When N ≤ K (Preserve Full Async) - **Direct Assignment**: Each controller gets dedicated worker immediately - **No Waiting**: `endShowLeds()` returns immediately after starting transmission - **Callback-Driven**: Use ESP-IDF `rmt_tx_event_callbacks_t::on_trans_done` for completion - **Zero Overhead**: Maintain current performance characteristics #### When N > K (Controlled Polling) - **Immediate Start**: First K controllers start immediately (async) - **Queue Remaining**: Controllers K+1 through N queue for workers - **Polling Strategy**: Use `delayMicroseconds(100)` polling for queued controllers - **Callback Coordination**: Workers signal completion via callbacks to start next queued controller ## ESP-IDF RMT5 Callback Integration ### Callback Registration Pattern ```cpp class RmtWorker { private: rmt_channel_handle_t mRmtChannel; RmtWorkerPool* mPool; static bool IRAM_ATTR onTransmissionComplete( rmt_channel_handle_t channel, const rmt_tx_done_event_data_t *edata, void *user_data) { RmtWorker* worker = static_cast(user_data); worker->handleTransmissionComplete(); return false; // No high-priority task woken } public: bool initialize() { // Create RMT channel rmt_tx_channel_config_t tx_config = { .gpio_num = mPin, .clk_src = RMT_CLK_SRC_DEFAULT, .resolution_hz = 10000000, // 10MHz .mem_block_symbols = 64, .trans_queue_depth = 4, }; if (rmt_new_tx_channel(&tx_config, &mRmtChannel) != ESP_OK) { return false; } // Register completion callback rmt_tx_event_callbacks_t callbacks = { .on_trans_done = onTransmissionComplete, }; return rmt_tx_register_event_callbacks(mRmtChannel, &callbacks, this) == ESP_OK; } void handleTransmissionComplete() { // Signal pool that this worker is available mPool->onWorkerComplete(this); } }; ``` ## Revised Worker Pool Architecture ### Async-Aware Worker Pool ```cpp class RmtWorkerPool { public: enum class DrawMode { ASYNC_ONLY, // N ≤ K: All controllers async MIXED_MODE // N > K: Some async, some polled }; void executeDrawCycle() { const int numControllers = mRegisteredControllers.size(); const int numWorkers = mAvailableWorkers.size(); if (numControllers <= numWorkers) { // ASYNC_ONLY mode - preserve full async behavior executeAsyncOnlyMode(); } else { // MIXED_MODE - async for first K, polling for rest executeMixedMode(); } } private: void executeAsyncOnlyMode() { // Start all controllers immediately - full async behavior preserved for (auto* controller : mRegisteredControllers) { RmtWorker* worker = acquireWorker(controller->getWorkerConfig()); startControllerAsync(controller, worker); } // Return immediately - no waiting! } void executeMixedMode() { // Start first K controllers immediately (async) int startedCount = 0; for (auto* controller : mRegisteredControllers) { if (startedCount < mAvailableWorkers.size()) { RmtWorker* worker = acquireWorker(controller->getWorkerConfig()); startControllerAsync(controller, worker); startedCount++; } else { // Queue remaining controllers mQueuedControllers.push_back(controller); } } // Poll for completion of queued controllers while (!mQueuedControllers.empty()) { delayMicroseconds(100); // Non-blocking poll interval // Callback-driven worker completion will process queue } } void onWorkerComplete(RmtWorker* worker) { // Called from ISR context via callback releaseWorker(worker); // Start next queued controller if available if (!mQueuedControllers.empty()) { RmtController5* nextController = mQueuedControllers.front(); mQueuedControllers.pop_front(); // Reconfigure worker and start transmission startControllerAsync(nextController, worker); } } }; ``` ### Modified RmtController5 for Async Preservation ```cpp class RmtController5 { public: void showPixels() { // This method MUST return immediately when N ≤ K // Only block when this specific controller is queued (N > K) RmtWorkerPool& pool = RmtWorkerPool::getInstance(); if (pool.canStartImmediately(this)) { // Async path - return immediately pool.startControllerImmediate(this); } else { // This controller is queued - must wait for worker pool.startControllerQueued(this); } } }; ``` ## Polling Strategy Details ### Microsecond Polling Pattern ```cpp void RmtWorkerPool::waitForQueuedControllers() { while (!mQueuedControllers.empty()) { // Non-blocking check for available workers if (hasAvailableWorker()) { processNextQueuedController(); } else { // Short delay to prevent busy-waiting delayMicroseconds(100); // 100μs polling interval } // Yield to other tasks periodically static uint32_t pollCount = 0; if (++pollCount % 50 == 0) { // Every 5ms (50 * 100μs) yield(); } } } ``` ### Callback-Driven Queue Processing ```cpp void RmtWorker::handleTransmissionComplete() { // Called from ISR context - keep minimal BaseType_t xHigherPriorityTaskWoken = pdFALSE; // Signal completion to pool xSemaphoreGiveFromISR(mPool->getCompletionSemaphore(), &xHigherPriorityTaskWoken); portYIELD_FROM_ISR(xHigherPriorityTaskWoken); } void RmtWorkerPool::processCompletionEvents() { // Called from main task context while (xSemaphoreTake(mCompletionSemaphore, 0) == pdTRUE) { // Process one completion event if (!mQueuedControllers.empty()) { RmtController5* nextController = mQueuedControllers.front(); mQueuedControllers.pop_front(); // Find available worker and start next transmission RmtWorker* worker = findAvailableWorker(); if (worker) { startControllerAsync(nextController, worker); } } } } ``` ## Benefits 1. **Async Preservation**: Full async behavior maintained when N ≤ K 2. **Scalability**: Support unlimited LED strips (within memory constraints) 3. **Backward Compatibility**: Existing code works unchanged 4. **Resource Efficiency**: Optimal use of limited RMT hardware 5. **Controlled Blocking**: Only blocks when specific controller is queued 6. **Callback Efficiency**: ISR-driven completion for minimal latency 7. **Polling Optimization**: 100μs intervals prevent busy-waiting ## Considerations 1. **Memory Usage**: Each controller needs persistent pixel buffer 2. **Latency**: Worker switching adds small overhead 3. **Complexity**: More complex than direct mapping 4. **Debugging**: Pool coordination harder to debug than direct control ## Migration Path The implementation maintains full backward compatibility. Existing code using `RmtController5` will automatically benefit from the worker pool without any changes required. ## CRITICAL: LED Buffer Transfer Analysis - CORRECTED FINDINGS ### ESP-IDF LED Buffer Management **Key Finding**: The ESP-IDF led_strip driver **supports external pixel buffers** and **buffer transfer IS possible through RMT channel recreation**. #### Current Buffer Architecture ```cpp typedef struct { led_strip_t base; rmt_channel_handle_t rmt_chan; rmt_encoder_handle_t strip_encoder; uint8_t *pixel_buf; // ← Buffer pointer (fixed at creation) bool pixel_buf_allocated_internally; // ← Ownership flag // ... other fields } led_strip_rmt_obj; ``` #### Buffer Creation Options ```cpp led_strip_config_t strip_config = { .strip_gpio_num = pin, .max_leds = led_count, .external_pixel_buf = external_buffer, // ← Can provide external buffer // ... other config }; ``` ### Buffer Transfer Solution - RMT Channel Recreation **✅ POSSIBLE**: Buffer transfer through worker reconfiguration - **Destroy existing RMT channel** when switching controllers - **Create new RMT channel** with different external buffer - **Worker pool manages appropriately-sized buffers** for each controller's requirements - **RMT channels always use external buffers** owned by the worker pool ### Critical Insight: Buffer Size Requirements **The Core Issue**: Different controllers need different buffer sizes: - **Controller A**: 100 RGB LEDs → 300 bytes buffer - **Controller B**: 200 RGBW LEDs → 800 bytes buffer - **RMT Channel**: Must be recreated with appropriate buffer size for each controller **ESP-IDF led_strip_rmt_obj Structure:** ```cpp typedef struct { led_strip_t base; rmt_channel_handle_t rmt_chan; rmt_encoder_handle_t strip_encoder; uint32_t strip_len; uint8_t bytes_per_pixel; led_color_component_format_t component_fmt; uint8_t *pixel_buf; // ← MUST be external and pool-managed bool pixel_buf_allocated_internally; // ← Always false for worker pool } led_strip_rmt_obj; ``` **Available Buffer APIs (Complete List):** - `led_strip_set_pixel()` - Writes to existing buffer - `led_strip_set_pixel_rgbw()` - Writes to existing buffer - `led_strip_clear()` - Zeros existing buffer - `led_strip_refresh_async()` - Transmits from existing buffer - **KEY**: `led_strip_del()` does NOT free external buffers (pixel_buf_allocated_internally = false) **Confirmed Solution**: Buffer transfer achieved through: 1. **Worker pool owns all RMT buffers** 2. **RMT channels destroyed/recreated** with appropriate buffer sizes 3. **External buffer management** prevents buffer deallocation during RMT destruction ### Buffer Transfer Solution: Worker Pool Buffer Management **CORRECTED APPROACH**: Worker pool manages all RMT buffers and handles buffer sizing for different controllers. #### Worker Pool Buffer Management Strategy ```cpp class RmtWorkerPool { private: struct WorkerState { IRmtStrip* strip; uint8_t* worker_buffer; // Pool-owned buffer for this worker size_t buffer_capacity; // Current buffer size WorkerConfig current_config; bool is_available; bool transmission_active; }; fl::vector mWorkers; // Buffer pool for different sizes fl::map> mBuffersBySize; public: bool assignWorkerToController(RmtController5* controller) { const WorkerConfig& config = controller->getWorkerConfig(); const size_t requiredBufferSize = config.led_count * (config.is_rgbw ? 4 : 3); // Find available worker WorkerState* worker = findAvailableWorker(); if (!worker) return false; // Get appropriately sized buffer from pool uint8_t* workerBuffer = acquireBuffer(requiredBufferSize); if (!workerBuffer) return false; // CRITICAL: Wait for any active transmission to complete if (worker->strip && worker->transmission_active) { worker->strip->waitDone(); worker->transmission_active = false; } // Destroy existing RMT channel if configuration changed if (worker->strip && (!configCompatible(worker->current_config, config) || worker->buffer_capacity < requiredBufferSize)) { delete worker->strip; // Destroy old RMT channel worker->strip = nullptr; // Release old buffer releaseBuffer(worker->worker_buffer, worker->buffer_capacity); } // Copy controller's pixel data to worker's buffer memcpy(workerBuffer, controller->getPixelBuffer(), controller->getBufferSize()); // Create new RMT channel with worker's external buffer if (!worker->strip) { worker->strip = IRmtStrip::CreateWithExternalBuffer( config.pin, config.led_count, config.is_rgbw, config.t0h, config.t0l, config.t1h, config.t1l, config.reset, workerBuffer, // Worker's buffer, not controller's buffer config.dma_mode ); if (!worker->strip) { releaseBuffer(workerBuffer, requiredBufferSize); return false; } } worker->worker_buffer = workerBuffer; worker->buffer_capacity = requiredBufferSize; worker->current_config = config; worker->is_available = false; // Start transmission worker->strip->drawAsync(); worker->transmission_active = true; return true; } void onWorkerComplete(WorkerState* worker) { // Transmission complete - RMT hardware done with buffer worker->transmission_active = false; if (!mQueuedControllers.empty()) { // Assign to next waiting controller RmtController5* nextController = mQueuedControllers.front(); mQueuedControllers.pop_front(); const WorkerConfig& nextConfig = nextController->getWorkerConfig(); const size_t nextBufferSize = nextConfig.led_count * (nextConfig.is_rgbw ? 4 : 3); if (worker->buffer_capacity >= nextBufferSize && configCompatible(worker->current_config, nextConfig)) { // OPTIMIZATION: Reuse existing buffer and RMT channel memcpy(worker->worker_buffer, nextController->getPixelBuffer(), nextController->getBufferSize()); worker->strip->drawAsync(); worker->transmission_active = true; } else { // RECONFIGURE: Need different buffer size or RMT configuration // Release current buffer releaseBuffer(worker->worker_buffer, worker->buffer_capacity); // Destroy RMT channel delete worker->strip; worker->strip = nullptr; // Get new appropriately-sized buffer worker->worker_buffer = acquireBuffer(nextBufferSize); if (!worker->worker_buffer) return; // Failed to get buffer // Copy next controller's data memcpy(worker->worker_buffer, nextController->getPixelBuffer(), nextController->getBufferSize()); // Create new RMT channel with new buffer worker->strip = IRmtStrip::CreateWithExternalBuffer( nextConfig.pin, nextConfig.led_count, nextConfig.is_rgbw, nextConfig.t0h, nextConfig.t0l, nextConfig.t1h, nextConfig.t1l, nextConfig.reset, worker->worker_buffer, // New worker buffer nextConfig.dma_mode ); worker->buffer_capacity = nextBufferSize; worker->current_config = nextConfig; // Start transmission worker->strip->drawAsync(); worker->transmission_active = true; } } else { // No waiting controllers - worker becomes available worker->is_available = true; // Keep buffer and RMT channel configured for potential reuse } } private: uint8_t* acquireBuffer(size_t size) { // Round up to nearest power of 2 for efficient pooling size_t poolSize = nextPowerOf2(size); auto& buffers = mBuffersBySize[poolSize]; if (!buffers.empty()) { uint8_t* buffer = buffers.back(); buffers.pop_back(); return buffer; } // Allocate new buffer return (uint8_t*)malloc(poolSize); } void releaseBuffer(uint8_t* buffer, size_t size) { size_t poolSize = nextPowerOf2(size); mBuffersBySize[poolSize].push_back(buffer); } }; ``` #### Simplified Controller Implementation ```cpp class RmtController5 { private: fl::vector mPixelBuffer; // Controller maintains its own data WorkerConfig mWorkerConfig; public: void loadPixelData(PixelIterator& pixels) { // Store pixel data in persistent buffer (unchanged) const int bytesPerPixel = mWorkerConfig.is_rgbw ? 4 : 3; const int bufferSize = pixels.size() * bytesPerPixel; mPixelBuffer.resize(bufferSize); // Load pixel data into our persistent buffer uint8_t* bufPtr = mPixelBuffer.data(); if (mWorkerConfig.is_rgbw) { while (pixels.has(1)) { uint8_t r, g, b, w; pixels.loadAndScaleRGBW(&r, &g, &b, &w); *bufPtr++ = r; *bufPtr++ = g; *bufPtr++ = b; *bufPtr++ = w; pixels.advanceData(); pixels.stepDithering(); } } else { while (pixels.has(1)) { uint8_t r, g, b; pixels.loadAndScaleRGB(&r, &g, &b); *bufPtr++ = r; *bufPtr++ = g; *bufPtr++ = b; pixels.advanceData(); pixels.stepDithering(); } } } void showPixels() { // Pool handles all buffer management internally RmtWorkerPool::getInstance().assignWorkerToController(this); } // Provide buffer access for pool management uint8_t* getPixelBuffer() { return mPixelBuffer.data(); } size_t getBufferSize() const { return mPixelBuffer.size(); } }; ``` ### Optimized Teardown Strategy **Key Insight**: RMT strip teardown should be **conditional** based on worker pool demand: #### When N ≤ K (No Queued Controllers) ```cpp void onWorkerComplete(RmtWorker* worker) { if (mQueuedControllers.empty()) { // NO TEARDOWN - keep worker configured and ready // Worker maintains its led_strip configuration // Optimizes for next frame if same controller used again releaseWorker(worker); } } ``` #### When N > K (Queued Controllers Waiting) ```cpp void onWorkerComplete(RmtWorker* worker) { if (!mQueuedControllers.empty()) { // IMMEDIATE TEARDOWN AND RECONFIGURATION // Next controller is waiting - reconfigure immediately RmtController5* nextController = mQueuedControllers.front(); mQueuedControllers.pop_front(); // This triggers teardown in reconfigure() startController(nextController, worker); } } ``` #### Buffer Change Requirement **CRITICAL**: Even with identical pin/LED count/timing configuration, **teardown is always required** when switching between different controller buffers: ```cpp // Controller A has buffer at 0x12345678 // Controller B has buffer at 0x87654321 // Even if both have same pin/count/timing: // - led_strip object MUST be recreated to use new buffer pointer // - ESP-IDF has no API to change pixel_buf after creation ``` ### Buffer Transfer Implementation Details #### Memory Safety - **Controller Ownership**: Each `RmtController5` owns its persistent buffer - **External Buffer Contract**: ESP-IDF won't free external buffers - **Worker Lifecycle**: Workers destroy/recreate led_strip objects as needed - **Buffer Validity**: Controllers must keep buffers valid during transmission #### Performance Considerations - **Reconfiguration Cost**: Creating new led_strip objects has overhead - **Buffer Copying**: No copying needed - workers use external buffers directly - **Memory Efficiency**: Only one buffer per controller (no duplication) #### Error Handling - **Reconfiguration Failures**: Handle led_strip creation failures gracefully - **Buffer Size Mismatches**: Validate buffer sizes during reconfiguration - **Transmission Errors**: Proper cleanup on transmission failures ### Buffer Transfer Summary - CORRECTED **✅ SOLUTION**: Worker pool buffer management with RMT channel recreation 1. **Worker pool owns all RMT buffers** sized appropriately for each controller 2. **Controllers maintain their own pixel data** in persistent buffers 3. **Buffer copying required** from controller buffer to worker buffer 4. **RMT channels destroyed/recreated** with appropriately-sized external buffers 5. **ESP-IDF transmits from worker's external buffer** (not controller's buffer) **✅ POSSIBLE**: Buffer transfer through worker reconfiguration - RMT channels can be destroyed and recreated with different external buffers - Worker pool manages buffer allocation and sizing - External buffers are not freed by ESP-IDF when RMT channels are destroyed **🔧 IMPLEMENTATION REQUIREMENTS**: - Worker pool must manage buffers of different sizes - RMT channel recreation required for buffer size changes - Transmission completion synchronization before reconfiguration - Buffer copying from controller to worker buffers ## CORRECTED CONCLUSIONS AND RECOMMENDATIONS ### Key Corrections to Original Analysis 1. **Buffer Transfer IS Possible**: The original "NOT POSSIBLE" assessment was incorrect. Buffer transfer can be achieved through RMT channel recreation with appropriately-sized external buffers. 2. **Worker Pool Must Manage Buffers**: The critical insight is that different controllers need different buffer sizes (RGB vs RGBW, different LED counts), so the worker pool must own and manage all RMT buffers. 3. **Buffer Copying Required**: Unlike the original zero-copy approach, buffer copying from controller to worker is necessary to handle different buffer size requirements. 4. **RMT Channel Recreation**: Workers must destroy and recreate RMT channels when switching between controllers with different requirements. ### Recommended Implementation Strategy **✅ RECOMMENDED**: Worker Pool Buffer Management - **Worker pool owns all RMT buffers** sized for different controller requirements - **RMT channels use external buffers** managed by the worker pool - **Buffer copying** from controller persistent buffers to worker buffers - **RMT channel recreation** when buffer size or configuration changes - **Transmission synchronization** to ensure safe reconfiguration ### Benefits of Corrected Approach - ✅ **Supports Variable Buffer Sizes**: Handles different LED counts and RGB/RGBW modes - ✅ **Proper Resource Management**: Worker pool manages buffer allocation/deallocation - ✅ **Clean Separation**: Controllers focus on pixel data, workers handle hardware - ✅ **Scalable Design**: Pool can optimize buffer reuse and minimize allocations - ✅ **Thread Safe**: Proper synchronization prevents buffer access conflicts ### Performance Considerations - ❌ **Buffer Copying Overhead**: Required due to different controller buffer sizes - ✅ **Buffer Reuse Optimization**: Pool can reuse buffers for compatible configurations - ✅ **Minimal RMT Recreation**: Only when buffer size or configuration changes - ✅ **Efficient Memory Usage**: Pool manages buffer sizes optimally ### Implementation Priority 1. **Phase 1**: Implement basic worker pool with buffer management 2. **Phase 2**: Add buffer size optimization and reuse logic 3. **Phase 3**: Implement transmission synchronization and callbacks 4. **Phase 4**: Add performance optimizations and error handling ## Future Enhancements 1. **Priority System**: Allow high-priority strips to get workers first 2. **Smart Batching**: Group compatible strips to minimize reconfiguration 3. **Dynamic Scaling**: Adjust worker count based on usage patterns 4. **Metrics**: Add performance monitoring and statistics 5. **Buffer Pool Optimization**: Advanced buffer size prediction and caching