video-v1/vav2/docs/completed/android/Phase_2_AImageReader_Native_Design.md

Phase 2: AImageReader Native API Implementation

작성일: 2025-10-14 목적: Java ImageReader → AImageReader Native API 교체로 성능 최적화 기반: Android NDK AImageReader API, Vulkan+Image+Tutorial.md API 요구사항: Android API 24+ (VavCore 타겟: API 26+) 상태: Ready for Implementation

---

1. 현재 구현 문제점

🔴 Problem 1: Sleep Workaround (5-9ms 블로킹)

// MediaCodecAsyncHandler.cpp:402
std::this_thread::sleep_for(std::chrono::milliseconds(5));  // BAD!

// MediaCodecAsyncHandler.cpp:419
for (int retry = 0; retry < 3; ++retry) {
    ahb = surface_manager->AcquireLatestImage();
    std::this_thread::sleep_for(std::chrono::milliseconds(2));  // BAD!
}

Impact: 30 FPS 대비 20-30% 레이턴시

🔴 Problem 2: JNI Overhead (10-20μs per frame)

// 현재: Java ImageReader + JNI Bridge
JNIEnv* env = GetJNIEnv();
jobject image = env->CallObjectMethod(reader, acquireMethod);        // JNI 1
jobject hardwareBuffer = env->CallObjectMethod(image, getMethod);    // JNI 2
AHardwareBuffer* ahb = AHardwareBuffer_fromHardwareBuffer(env, hb);  // JNI 3

Impact: 3x JNI calls + Java GC pressure

---

2. 목표: AImageReader Native API

성능 개선

지표	Before (Java)	After (Native)	개선
Sleep blocking	5-9ms	0ms	완전 제거
JNI calls	3 per frame	0	완전 제거
Frame latency	10-15ms	2-5ms	70% 감소
Code complexity	200 lines (Java+JNI)	50 lines (C++)	75% 감소

아키텍처

Before (Java ImageReader):

MediaCodec → [JNI] → Java ImageReader → [JNI] → C++ Handler
              ↑ 3 calls ↑

After (AImageReader Native):

MediaCodec → AImageReader → C++ Handler (Direct callback)
              ↑ 0 JNI ↑

---

3. 구현 설계

3.1 MediaCodecSurfaceManager 헤더 수정

파일: MediaCodecSurfaceManager.h

#pragma once

#ifdef ANDROID
#include <media/NdkImageReader.h>  // ← NEW: AImageReader API
#include <android/native_window.h>
#include <android/hardware_buffer.h>

class MediaCodecSurfaceManager {
public:
    // ImageReader setup (CHANGED: Native API)
    bool SetupImageReader(uint32_t width, uint32_t height);

    // Image acquisition (CHANGED: AImage instead of jobject)
    AHardwareBuffer* AcquireLatestImage();
    void ReleaseImage();

    // NEW: Native callback for OnImageAvailable
    void OnImageAvailableCallback(AImageReader* reader);

    // Current image tracking
    bool HasCurrentImage() const { return m_current_image != nullptr; }
    AHardwareBuffer* GetCurrentAHardwareBuffer() const { return m_current_ahardware_buffer; }

private:
    // REMOVED: jobject m_image_reader (Java)
    // REMOVED: jobject m_current_image (Java)

    // NEW: Native types
    AImageReader* m_image_reader;           // Native ImageReader
    AImage* m_current_image;                // Native Image
    AHardwareBuffer* m_current_ahardware_buffer;

    // Video dimensions
    uint32_t m_video_width;
    uint32_t m_video_height;

    // Synchronization for callback
    std::mutex m_image_mutex;
    std::condition_variable m_image_cv;
    std::atomic<bool> m_image_available{false};

    // Static callback (for AImageReader_setImageListener)
    static void OnImageAvailableStatic(void* context, AImageReader* reader);
};

3.2 SetupImageReader 구현 (Native)

파일: MediaCodecSurfaceManager.cpp

#include <media/NdkImageReader.h>

bool MediaCodecSurfaceManager::SetupImageReader(uint32_t width, uint32_t height) {
    m_video_width = width;
    m_video_height = height;

    LogInfo("SetupImageReader: Creating AImageReader (Native) " +
            std::to_string(width) + "x" + std::to_string(height));

    // REMOVED: All JNI code
    // REMOVED: Java ImageReader creation

    // NEW: Create AImageReader (Native)
    media_status_t status = AImageReader_new(
        width,
        height,
        AIMAGE_FORMAT_PRIVATE,  // MediaCodec output format
        3,                      // maxImages (same as Java ImageReader)
        &m_image_reader
    );

    if (status != AMEDIA_OK || !m_image_reader) {
        LogError("Failed to create AImageReader: " + std::to_string(status));
        return false;
    }

    LogInfo("AImageReader created successfully");

    // NEW: Set native image listener (NO JAVA!)
    AImageReader_ImageListener listener{
        .context = this,
        .onImageAvailable = OnImageAvailableStatic
    };

    status = AImageReader_setImageListener(m_image_reader, &listener);
    if (status != AMEDIA_OK) {
        LogError("Failed to set image listener: " + std::to_string(status));
        AImageReader_delete(m_image_reader);
        m_image_reader = nullptr;
        return false;
    }

    LogInfo("Image listener registered (native callback)");

    // NEW: Get ANativeWindow from AImageReader (NO JAVA!)
    status = AImageReader_getWindow(m_image_reader, &m_surface);
    if (status != AMEDIA_OK || !m_surface) {
        LogError("Failed to get window from AImageReader: " + std::to_string(status));
        AImageReader_delete(m_image_reader);
        m_image_reader = nullptr;
        return false;
    }

    LogInfo("ImageReader surface obtained: " +
            std::to_string(reinterpret_cast<uintptr_t>(m_surface)));

    return true;
}

3.3 Native Callback 구현

파일: MediaCodecSurfaceManager.cpp

// Static callback (called by AImageReader)
void MediaCodecSurfaceManager::OnImageAvailableStatic(void* context, AImageReader* reader) {
    auto* manager = static_cast<MediaCodecSurfaceManager*>(context);
    if (manager) {
        manager->OnImageAvailableCallback(reader);
    }
}

// Instance callback
void MediaCodecSurfaceManager::OnImageAvailableCallback(AImageReader* reader) {
    LogInfo("OnImageAvailableCallback: ENTRY (Native callback)");

    // This callback is invoked on a dedicated AImageReader thread
    // It's safe to call AImageReader_* and AImage_* methods here

    std::lock_guard<std::mutex> lock(m_image_mutex);

    // Acquire latest image (NO JNI!)
    AImage* image = nullptr;
    media_status_t status = AImageReader_acquireLatestImage(reader, &image);

    if (status != AMEDIA_OK || !image) {
        LogWarning("OnImageAvailableCallback: Failed to acquire image: " +
                  std::to_string(status));
        return;
    }

    LogInfo("OnImageAvailableCallback: Image acquired successfully");

    // Get AHardwareBuffer (Direct pointer access - NO JNI!)
    AHardwareBuffer* ahb = nullptr;
    status = AImage_getHardwareBuffer(image, &ahb);

    if (status != AMEDIA_OK || !ahb) {
        LogError("OnImageAvailableCallback: Failed to get AHardwareBuffer: " +
                std::to_string(status));
        AImage_delete(image);
        return;
    }

    // Acquire reference to AHardwareBuffer
    AHardwareBuffer_acquire(ahb);

    // Store current image
    if (m_current_image) {
        AImage_delete(m_current_image);
        m_current_image = nullptr;
    }

    if (m_current_ahardware_buffer) {
        AHardwareBuffer_release(m_current_ahardware_buffer);
        m_current_ahardware_buffer = nullptr;
    }

    m_current_image = image;
    m_current_ahardware_buffer = ahb;

    // Signal waiting thread
    m_image_available = true;
    m_image_cv.notify_one();

    LogInfo("OnImageAvailableCallback: EXIT - Image stored and signaled");
}

3.4 AcquireLatestImage 수정 (Wait for Callback)

파일: MediaCodecSurfaceManager.cpp

AHardwareBuffer* MediaCodecSurfaceManager::AcquireLatestImage() {
    // REMOVED: JNI calls
    // REMOVED: Java ImageReader.acquireLatestImage()

    // NEW: Wait for OnImageAvailableCallback
    std::unique_lock<std::mutex> lock(m_image_mutex);

    // Wait for image with timeout (100ms)
    bool image_ready = m_image_cv.wait_for(
        lock,
        std::chrono::milliseconds(100),
        [this] { return m_image_available.load(); }
    );

    if (!image_ready || !m_current_ahardware_buffer) {
        LogWarning("AcquireLatestImage: Timeout waiting for image");
        return nullptr;
    }

    // Reset flag
    m_image_available = false;

    LogInfo("AcquireLatestImage: Returning cached AHardwareBuffer");
    return m_current_ahardware_buffer;
}

3.5 ReleaseImage 수정 (Native)

파일: MediaCodecSurfaceManager.cpp

void MediaCodecSurfaceManager::ReleaseImage() {
    // REMOVED: JNI calls
    // REMOVED: Java Image.close()

    // NEW: Native AImage cleanup
    std::lock_guard<std::mutex> lock(m_image_mutex);

    if (m_current_image) {
        AImage_delete(m_current_image);  // ← Native API
        m_current_image = nullptr;
        LogInfo("AImage deleted (native)");
    }

    if (m_current_ahardware_buffer) {
        AHardwareBuffer_release(m_current_ahardware_buffer);
        m_current_ahardware_buffer = nullptr;
        LogInfo("AHardwareBuffer released");
    }
}

3.6 Cleanup 수정

파일: MediaCodecSurfaceManager.cpp

void MediaCodecSurfaceManager::Cleanup() {
    // Release current image
    ReleaseImage();

    // REMOVED: JNI cleanup
    // REMOVED: Java ImageReader deletion

    // NEW: Native AImageReader cleanup
    if (m_image_reader) {
        AImageReader_delete(m_image_reader);
        m_image_reader = nullptr;
        LogInfo("AImageReader deleted (native)");
    }

    // Rest of cleanup...
}

3.7 ProcessAsyncOutputFrame 수정 (Remove Sleep!)

파일: MediaCodecAsyncHandler.cpp

bool MediaCodecAsyncHandler::ProcessAsyncOutputFrame(
    int32_t output_index,
    AMediaCodecBufferInfo* buffer_info,
    VideoFrame& output_frame)
{
    // Phase 1: GPU-synchronized release (unchanged)
    MediaCodecSurfaceManager* surface_manager = m_decoder->GetSurfaceManager();
    if (surface_manager && surface_manager->HasCurrentImage()) {
        void* vk_device = surface_manager->GetVulkanDevice();
        if (vk_device) {
            surface_manager->ReleaseImageAfterGPU(static_cast<VkDevice>(vk_device));
        }
    }

    // Release MediaCodec buffer to ImageReader surface
    media_status_t status = AMediaCodec_releaseOutputBuffer(m_codec, output_index, true);
    if (status != AMEDIA_OK) {
        return false;
    }

    // ====== PHASE 2 CHANGE: Wait for Native Callback ======
    // REMOVED: std::this_thread::sleep_for(std::chrono::milliseconds(5));
    // REMOVED: Retry loop with sleep(2ms)

    // OnImageAvailableCallback will be triggered asynchronously
    // AcquireLatestImage() now waits for callback with condition variable
    // ====== END PHASE 2 CHANGE ======

    // Acquire AHardwareBuffer (callback already happened)
    AHardwareBuffer* ahb = surface_manager->AcquireLatestImage();
    if (!ahb) {
        LogError("ProcessAsyncOutputFrame: Failed to acquire image");
        return false;
    }

    // Rest of code unchanged (Vulkan image creation, etc.)
    // ...

    return true;
}

---

4. 구현 단계

Phase 2.1: MediaCodecSurfaceManager 리팩토링 (2-3시간)

작업 순서:

1. ✅ MediaCodecSurfaceManager.h - AImageReader 멤버 추가
2. ✅ SetupImageReader() 재구현 (Native API)
3. ✅ OnImageAvailableCallback() 구현
4. ✅ AcquireLatestImage() 수정 (condition variable wait)
5. ✅ ReleaseImage() 수정 (AImage_delete)
6. ✅ Cleanup() 수정 (AImageReader_delete)

수정 파일:

• MediaCodecSurfaceManager.h - 헤더 수정
• MediaCodecSurfaceManager.cpp - 전체 재구현

Phase 2.2: ProcessAsyncOutputFrame 단순화 (30분)

작업 순서:

1. ✅ sleep_for(5ms) 제거
2. ✅ Retry loop 제거
3. ✅ 로직 단순화

수정 파일:

• MediaCodecAsyncHandler.cpp - ProcessAsyncOutputFrame 수정

Phase 2.3: 빌드 및 테스트 (1-2시간)

테스트 항목:

1. ✅ Android ARM64 빌드 성공
2. ✅ AImageReader 생성 확인 (로그)
3. ✅ Native callback 호출 확인 (로그)
4. ✅ 30 FPS 달성
5. ✅ Sleep 제거 확인 (로그에서 "sleep" 검색)
6. ✅ 레이턴시 측정 (Phase 1 대비)

---

5. 제거할 코드

❌ 삭제할 Java 파일

• ImageReaderCallback.java - 생성했지만 불필요

❌ 제거할 JNI 코드

• MediaCodecSurfaceManager.cpp:
  • GetJNIEnv() 호출 (ImageReader 관련만)
  • env->FindClass("android/media/ImageReader")
  • env->CallObjectMethod() 호출들
  • env->NewGlobalRef() / DeleteGlobalRef() (Image 관련)
  • AHardwareBuffer_fromHardwareBuffer() 변환

❌ 제거할 Workaround

• std::this_thread::sleep_for(std::chrono::milliseconds(5))
• std::this_thread::sleep_for(std::chrono::milliseconds(2))
• Retry loop (3회 반복)

---

6. 성능 예상

Before (Phase 1 - Java ImageReader)

releaseOutputBuffer
  → sleep(5ms)
  → JNI acquireLatestImage (1-5μs)
  → JNI getHardwareBuffer (1-5μs)
  → JNI fromHardwareBuffer (5-10μs)
  → [retry loop: sleep(2ms) x2]

Total: 9-15ms per frame

After (Phase 2 - AImageReader Native)

releaseOutputBuffer
  → OnImageAvailableCallback (async, <1ms)
  → AcquireLatestImage (wait on condition_variable)
  → Direct pointer access (<1μs)

Total: 2-5ms per frame

개선:

• ✅ 레이턴시: 60-70% 감소 (9ms → 3ms)
• ✅ JNI overhead: 100% 제거
• ✅ Java GC pressure: 100% 제거
• ✅ 코드 복잡도: 75% 감소

---

7. 검증 체크리스트

빌드 검증

• Android ARM64 빌드 성공
• AImageReader API 링크 확인 (libmediandk.so)
• No JNI errors

기능 검증

• AImageReader 생성 성공 (로그)
• Native callback 호출 확인 (로그: "OnImageAvailableCallback")
• AImage_getHardwareBuffer 성공
• Vulkan image 생성 성공

성능 검증

• 30 FPS 이상 달성
• Sleep 완전 제거 확인 (grep "sleep")
• 레이턴시 < 5ms (로그 타임스탬프 측정)
• CPU 사용률 감소

안정성 검증

• 10분 이상 재생 테스트
• 메모리 누수 확인 (AImage_delete, AHardwareBuffer_release)
• 크래시 없음 확인

---

8. Rollback 계획

Phase 2 구현 중 문제 발생 시:

1. Git revert: Phase 1 커밋으로 즉시 복구
2. Phase 1은 검증됨: 프로덕션 사용 가능

---

9. 다음 단계

Phase 2 완료 후

1. ✅ 성능 벤치마크 측정 (Phase 1 vs Phase 2)
2. ✅ 문서 업데이트 (COMPLETED_PROJECTS.md)
3. ✅ Phase 1 설계서 보관 (old design reference)

향후 최적화 (Optional - Phase 3)

• SyncFence 통합 (API 33+)
• Adaptive buffer size (2-5 frames)
• Frame dropping strategy

---

문서 상태: ✅ Ready for Implementation 예상 구현 시간: 3-5시간 우선순위: 🟢 High (10-20배 성능 향상 예상) Java/JNI 의존성: ❌ None (100% Native C++)