video-v1/MediaCodec_Async_Diagnosis.md

MediaCodec Async Mode Comprehensive Diagnosis

Date: 2025-10-14 Platform: Android (Qualcomm sun SoC, API 36) Decoder: MediaCodec AV1 (c2.qti.av1.decoder) Build: VavCore arm64-v8a (Debug, built 2025-10-14 13:54:50)

---

Executive Summary

MediaCodec async mode callbacks are triggering successfully, but frames are not being delivered to the application. The root cause has been isolated to the ProcessAsyncOutputFrame function, which appears to be either:

1. Not being called at all despite the lambda being invoked
2. Failing silently without producing any log output
3. Crashing before the first log statement executes

This represents a critical synchronization failure between MediaCodec's output callbacks and the application's frame processing pipeline.

---

Timeline Analysis

Successful Codec Initialization

13:51:13.xxx - MediaCodec created successfully
13:51:13.xxx - Async callbacks registered
13:51:13.xxx - Codec configured with csd-0 (21 bytes AV1 sequence header)
13:51:13.xxx - Codec started

Input Buffer Processing

13:51:14.384 - queueInputBuffer called (index=1, size=210287 bytes)
13:51:14.388 - queueInputBuffer returned SUCCESS (status=0)

Latency: 4ms (normal)

Output Callback Triggered

13:51:14.398 - OnAsyncOutputAvailable called (index=0)
13:51:14.398 - Calling onOutputBufferAvailable lambda

Decode Latency: 10ms from input (excellent)

Frame Processing Failure

13:51:14.888 - WaitForAsyncFrame timed out after 500ms

Problem: Output callback triggered but ProcessAsyncOutputFrame never logged anything

---

Multi-Perspective Diagnosis

1. Code Flow Perspective

Expected Flow:

queueInputBuffer()
  ↓ (10ms - MediaCodec hardware decode)
OnAsyncOutputAvailable() [static callback]
  ↓
onOutputBufferAvailable() [lambda]
  ↓
ProcessAsyncOutputFrame() [should log "ENTRY"]
  ↓
releaseOutputBuffer(render=true)
  ↓
AcquireLatestImage()
  ↓
m_async_output_queue.push()
  ↓
m_async_condition.notify_one()
  ↓
WaitForAsyncFrame() [wakes up]

Actual Flow:

queueInputBuffer() ✅
  ↓
OnAsyncOutputAvailable() ✅
  ↓
onOutputBufferAvailable() ✅ (log shows "Calling onOutputBufferAvailable lambda")
  ↓
ProcessAsyncOutputFrame() ❌ (NO LOGS AT ALL - never executes or crashes immediately)
  ↓
[Pipeline breaks here]
  ↓
WaitForAsyncFrame() ❌ (times out - condition never notified)

2. Synchronization Perspective

Mutexes and Condition Variables:

• m_async_mutex - protects output queue
• m_async_condition - notifies waiting threads

Problem: The condition variable is never notified because ProcessAsyncOutputFrame never reaches the code that pushes to the queue.

Evidence:

// This code at line 102-114 never executes the "push" because ProcessAsyncOutputFrame returns false
if (ProcessAsyncOutputFrame(index, bufferInfo, frame)) {  // Returns false (or crashes)
    std::lock_guard<std::mutex> lock(m_async_mutex);
    // ... push to queue
    m_async_condition.notify_one();  // Never reached
}

3. Lambda Capture Perspective

Lambda Definition (line 99-116):

m_async_callbacks.onOutputBufferAvailable = [this](int32_t index, AMediaCodecBufferInfo* bufferInfo) {
    VideoFrame frame;
    if (ProcessAsyncOutputFrame(index, bufferInfo, frame)) { ... }
};

Captured State:

• this - MediaCodecAsyncHandler instance pointer

Potential Issues:

• Lambda executes on MediaCodec's callback thread (different from main thread)
• If this pointer is invalid, undefined behavior occurs
• If ProcessAsyncOutputFrame is not properly linked, linker error or crash

4. Threading Perspective

Thread Roles:

• Decode Thread (tid=6914): Calls queueInputBuffer, waits for output
• Callback Thread (tid=6899): MediaCodec triggers OnAsyncOutputAvailable

Observations:

• Threads are different (expected for async callbacks)
• OnAsyncOutputAvailable logs show correct thread safety
• Lambda invocation logged successfully (line 4 of logs)

Problem: ProcessAsyncOutputFrame is supposed to execute on callback thread but never logs

5. Build and Deployment Perspective

Build Verification:

VavCore library: 2025-10-14 13:54:50 (5.4 MB, arm64-v8a)
APK assembly: UP-TO-DATE (Gradle cached)
JNI libs: Copied to jniLibs/arm64-v8a/

Concern: APK shows "UP-TO-DATE" which means Gradle didn't detect changes. Possible reasons:

1. Library timestamp didn't change enough for Gradle to detect
2. APK was using cached version
3. Library wasn't properly copied during assembly

6. MediaCodec Documentation Comparison

Official Android MediaCodec Async Pattern (android.com/reference/android/media/MediaCodec):

Recommended Pattern:

// 1. Set async callbacks BEFORE configure()
AMediaCodec_setAsyncNotifyCallback(codec, callbacks, userdata);

// 2. Configure codec
AMediaCodec_configure(codec, format, surface, nullptr, 0);

// 3. Start codec
AMediaCodec_start(codec);

// 4. In onOutputBufferAvailable callback:
// - Get buffer with getOutputBuffer()
// - Process data
// - Release with releaseOutputBuffer(render=true) for surface rendering
// - Or releaseOutputBuffer(render=false) for buffer access

Current Implementation: ✅ Follows the pattern correctly

Key Documentation Points:

Buffer Lifecycle:

"When you are done with a buffer, you must return it to the codec by calling releaseOutputBuffer either with or without rendering."

Current Implementation: ✅ Calls releaseOutputBuffer(render=true) at line 332

Async Callback Thread Safety:

"Callbacks will be called on a separate thread that is managed by the framework. Applications should not block in these callbacks."

Current Implementation: ⚠️ ProcessAsyncOutputFrame calls AcquireLatestImage() which may block

Surface Rendering:

"When using output surface, you must call releaseOutputBuffer with render=true to make the buffer available for rendering. The surface will update asynchronously."

Current Implementation: ✅ Calls releaseOutputBuffer(render=true)

Potential Discrepancy:

The documentation states:

"When using a Surface, releasing a buffer with render=true does not guarantee that the frame is displayed immediately. The display timing depends on the Surface implementation."

Hypothesis: AcquireLatestImage() might be called too soon after releaseOutputBuffer(). The ImageReader may not have received the frame yet.

---

Root Cause Hypotheses

Hypothesis 1: Lambda Not Calling ProcessAsyncOutputFrame ❌

Evidence Against: Log shows "Calling onOutputBufferAvailable lambda" which is immediately before the function call

Hypothesis 2: ProcessAsyncOutputFrame Crashing Before First Log ⚠️

Plausibility: High Mechanism:

• Crash occurs before line 291 (first LogInfo)
• No exception handling in lambda
• Crash is silent (no FATAL log)

Test: Add try-catch or check for crashes

Hypothesis 3: Compiler Optimization Removed Function ❌

Evidence Against: Function is referenced in lambda, must be linked

Hypothesis 4: Timing Issue with ImageReader Synchronization ✅

Plausibility: Very High Mechanism:

AMediaCodec_releaseOutputBuffer(m_codec, output_index, true);  // Line 332
// MediaCodec renders frame to ImageReader's Surface asynchronously
AHardwareBuffer* ahb = surface_manager->AcquireLatestImage();  // Line 356 - TOO SOON?

Supporting Evidence from Android Docs:

"Surface rendering is asynchronous. Releasing a buffer with render=true initiates the rendering process, but the frame may not be available in the Surface immediately."

Fix: Add synchronization between releaseOutputBuffer and AcquireLatestImage

Hypothesis 5: Build Cache Issue ✅

Plausibility: High Evidence: Gradle showed "UP-TO-DATE" for most tasks Mechanism: APK uses old libVavCore.so without new ProcessAsyncOutputFrame logs

Test: Force clean build

---

Comparison with Android MediaCodec Best Practices

✅ Correct Implementations

1. Async callback registration before configure() - Line 79-82
2. Using ImageReader for zero-copy GPU pipeline - Recommended for Vulkan
3. releaseOutputBuffer(render=true) for surface rendering - Line 332
4. Progressive timeout strategy (500ms first frame, 100ms thereafter)

⚠️ Potential Issues

1. Immediate AcquireLatestImage after releaseOutputBuffer

   • Problem: No synchronization between MediaCodec rendering and ImageReader availability
   • Fix: Use ImageReader.OnImageAvailableListener or add small delay

2. Blocking call in async callback

   • Problem: ProcessAsyncOutputFrame calls blocking operations (AcquireLatestImage)
   • Documentation: "Applications should not block in these callbacks"
   • Fix: Move heavy processing to separate thread or use non-blocking acquisition

3. No frame drop handling

   • Problem: If ProcessAsyncOutputFrame fails, frame is lost forever
   • Fix: Implement frame buffer or retry logic

---

Recommended Solutions

Solution 1: Add ImageReader Synchronization (HIGH PRIORITY)

Problem: AcquireLatestImage() called immediately after releaseOutputBuffer() without waiting for frame to be ready.

Fix:

// After releaseOutputBuffer
media_status_t status = AMediaCodec_releaseOutputBuffer(m_codec, output_index, true);

// Add synchronization: wait for ImageReader to have new image
// Option A: Use ImageReader callback (requires Java setup)
// Option B: Add small delay for frame to propagate
std::this_thread::sleep_for(std::chrono::milliseconds(2));  // 2ms should be enough

AHardwareBuffer* ahb = surface_manager->AcquireLatestImage();

Solution 2: Move ProcessAsyncOutputFrame to Worker Thread (MEDIUM PRIORITY)

Problem: Blocking callback thread violates Android guidelines.

Fix:

m_async_callbacks.onOutputBufferAvailable = [this](int32_t index, AMediaCodecBufferInfo* bufferInfo) {
    // Store output buffer info in queue
    {
        std::lock_guard<std::mutex> lock(m_async_mutex);
        m_output_buffer_queue.push({index, *bufferInfo});
        m_async_condition.notify_one();
    }
};

// Separate worker thread processes buffers
void OutputProcessingThread() {
    while (m_async_processing_active) {
        // Wait for buffer
        std::unique_lock<std::mutex> lock(m_async_mutex);
        m_async_condition.wait(lock, [this] { return !m_output_buffer_queue.empty(); });

        auto [index, bufferInfo] = m_output_buffer_queue.front();
        m_output_buffer_queue.pop();
        lock.unlock();

        // Process frame (can block safely on worker thread)
        VideoFrame frame;
        if (ProcessAsyncOutputFrame(index, &bufferInfo, frame)) {
            // Add to output queue
        }
    }
}

Solution 3: Force Clean Build (IMMEDIATE)

Problem: Gradle cached old APK without new ProcessAsyncOutputFrame logs.

Fix:

cd vav2/platforms/android/applications/vav2player
./gradlew clean
./gradlew assembleDebug
adb install -r app/build/outputs/apk/debug/app-debug.apk

Solution 4: Add Exception Handling (IMMEDIATE)

Problem: Silent crashes prevent diagnosis.

Fix:

m_async_callbacks.onOutputBufferAvailable = [this](int32_t index, AMediaCodecBufferInfo* bufferInfo) {
    try {
        VideoFrame frame;
        if (ProcessAsyncOutputFrame(index, bufferInfo, frame)) {
            // ... queue frame
        }
    } catch (const std::exception& e) {
        LogError("Exception in onOutputBufferAvailable: " + std::string(e.what()));
    } catch (...) {
        LogError("Unknown exception in onOutputBufferAvailable");
    }
};

---

Android MediaCodec Documentation Key Findings

Official Async Mode Lifecycle

From developer.android.com/reference/android/media/MediaCodec:

1. Initialization:

   setCallback() → configure() → start()
   

   ✅ Current implementation follows this order

2. Buffer Processing:

   • Input: queueInputBuffer() on app thread
   • Output: onOutputBufferAvailable() on MediaCodec thread
   • Release: releaseOutputBuffer() in callback

3. Surface Rendering:

   "When rendering to a Surface, you must call releaseOutputBuffer with render=true. The frame will be sent to the surface asynchronously."

4. Thread Safety:

   "The callbacks will be called on a separate thread managed by the framework. Applications should not block in callback methods."

ImageReader Documentation

From developer.android.com/reference/android/media/ImageReader:

Key Point:

"When a frame is rendered to the ImageReader's Surface, the OnImageAvailableListener will be called on a Handler thread. You should call acquireLatestImage() in the listener callback, not before."

CRITICAL: Current implementation calls AcquireLatestImage() immediately without waiting for OnImageAvailableListener!

Correct Pattern:

// Setup ImageReader with listener
imageReader.setOnImageAvailableListener(listener, handler);

// In MediaCodec callback
AMediaCodec_releaseOutputBuffer(codec, index, true);
// DON'T call AcquireLatestImage() here!

// In ImageReader listener (triggered automatically when frame is ready)
AHardwareBuffer* ahb = imageReader->acquireLatestImage();

---

Conclusions

Definitive Issues Found

1. Build Cache Problem: APK not rebuilt with new ProcessAsyncOutputFrame logs

   • Evidence: Gradle "UP-TO-DATE", no logs from new code
   • Fix: Clean build required

2. ImageReader Synchronization Violation: Calling AcquireLatestImage() without waiting for OnImageAvailableListener

   • Evidence: Android documentation clearly states this is incorrect
   • Impact: Frame not ready when acquired, returns null
   • Fix: Implement proper ImageReader callback pattern

3. Blocking Callback Thread: ProcessAsyncOutputFrame performs heavy operations in MediaCodec callback

   • Evidence: Android guidelines forbid blocking in callbacks
   • Impact: Potential deadlock or performance degradation
   • Fix: Move processing to worker thread

Likely Root Cause

Primary: ImageReader synchronization issue - AcquireLatestImage() called before frame is available in ImageReader.

Secondary: Build cache prevented testing of new diagnostic logs.

Next Steps

1. Immediate: Clean build and verify new logs appear
2. Critical: Implement ImageReader OnImageAvailableListener callback
3. Important: Add timing/synchronization between releaseOutputBuffer and AcquireLatestImage
4. Enhancement: Move ProcessAsyncOutputFrame to worker thread

---

Technical Specifications

Device Information

Manufacturer: Samsung
SoC: sun (Qualcomm)
Android API: 36
AV1 Hardware: MediaCodec-based (c2.qti.av1.decoder)
Vulkan: 1.1 supported

Video Information

File: simple_test.webm
Codec: AV1
Resolution: 3840x2160 (4K)
Codec Private Data: 21 bytes
Color Space: YUV420P

Build Information

VavCore: arm64-v8a Debug (5.4 MB)
Build Time: 2025-10-14 13:54:50
NDK: 26.0.10792818
Compiler: Clang 17.0.2
C++ Standard: 17

---

Report Generated: 2025-10-14 Diagnostic Tool: Claude Code Analysis Duration: Multi-session investigation