video-v1/vav2/docs/completed/android/Phase_2_AImageReader_Native_Design.md

# Phase 2: AImageReader Native API Implementation

**작성일:** 2025-10-14
**목적:** Java ImageReader → AImageReader Native API 교체로 성능 최적화
**기반:** Android NDK AImageReader API, Vulkan+Image+Tutorial.md
**API 요구사항:** Android API 24+ (VavCore 타겟: API 26+)
**상태:** Ready for Implementation

---

## 1. 현재 구현 문제점

### 🔴 Problem 1: Sleep Workaround (5-9ms 블로킹)

```cpp
// MediaCodecAsyncHandler.cpp:402
std::this_thread::sleep_for(std::chrono::milliseconds(5));  // BAD!

// MediaCodecAsyncHandler.cpp:419
for (int retry = 0; retry < 3; ++retry) {
    ahb = surface_manager->AcquireLatestImage();
    std::this_thread::sleep_for(std::chrono::milliseconds(2));  // BAD!
}
```

**Impact:** 30 FPS 대비 20-30% 레이턴시

### 🔴 Problem 2: JNI Overhead (10-20μs per frame)

```cpp
// 현재: Java ImageReader + JNI Bridge
JNIEnv* env = GetJNIEnv();
jobject image = env->CallObjectMethod(reader, acquireMethod);        // JNI 1
jobject hardwareBuffer = env->CallObjectMethod(image, getMethod);    // JNI 2
AHardwareBuffer* ahb = AHardwareBuffer_fromHardwareBuffer(env, hb);  // JNI 3
```

**Impact:** 3x JNI calls + Java GC pressure

---

## 2. 목표: AImageReader Native API

### 성능 개선

| 지표 | Before (Java) | After (Native) | 개선 |
|------|---------------|----------------|------|
| Sleep blocking | 5-9ms | 0ms | **완전 제거** |
| JNI calls | 3 per frame | 0 | **완전 제거** |
| Frame latency | 10-15ms | 2-5ms | **70% 감소** |
| Code complexity | 200 lines (Java+JNI) | 50 lines (C++) | **75% 감소** |

### 아키텍처

**Before (Java ImageReader):**
```
MediaCodec → [JNI] → Java ImageReader → [JNI] → C++ Handler
              ↑ 3 calls ↑
```

**After (AImageReader Native):**
```
MediaCodec → AImageReader → C++ Handler (Direct callback)
              ↑ 0 JNI ↑
```

---

## 3. 구현 설계

### 3.1 MediaCodecSurfaceManager 헤더 수정

**파일:** `MediaCodecSurfaceManager.h`

```cpp
#pragma once

#ifdef ANDROID
#include <media/NdkImageReader.h>  // ← NEW: AImageReader API
#include <android/native_window.h>
#include <android/hardware_buffer.h>

class MediaCodecSurfaceManager {
public:
    // ImageReader setup (CHANGED: Native API)
    bool SetupImageReader(uint32_t width, uint32_t height);

    // Image acquisition (CHANGED: AImage instead of jobject)
    AHardwareBuffer* AcquireLatestImage();
    void ReleaseImage();

    // NEW: Native callback for OnImageAvailable
    void OnImageAvailableCallback(AImageReader* reader);

    // Current image tracking
    bool HasCurrentImage() const { return m_current_image != nullptr; }
    AHardwareBuffer* GetCurrentAHardwareBuffer() const { return m_current_ahardware_buffer; }

private:
    // REMOVED: jobject m_image_reader (Java)
    // REMOVED: jobject m_current_image (Java)

    // NEW: Native types
    AImageReader* m_image_reader;           // Native ImageReader
    AImage* m_current_image;                // Native Image
    AHardwareBuffer* m_current_ahardware_buffer;

    // Video dimensions
    uint32_t m_video_width;
    uint32_t m_video_height;

    // Synchronization for callback
    std::mutex m_image_mutex;
    std::condition_variable m_image_cv;
    std::atomic<bool> m_image_available{false};

    // Static callback (for AImageReader_setImageListener)
    static void OnImageAvailableStatic(void* context, AImageReader* reader);
};
```

### 3.2 SetupImageReader 구현 (Native)

**파일:** `MediaCodecSurfaceManager.cpp`

```cpp
#include <media/NdkImageReader.h>

bool MediaCodecSurfaceManager::SetupImageReader(uint32_t width, uint32_t height) {
    m_video_width = width;
    m_video_height = height;

    LogInfo("SetupImageReader: Creating AImageReader (Native) " +
            std::to_string(width) + "x" + std::to_string(height));

    // REMOVED: All JNI code
    // REMOVED: Java ImageReader creation

    // NEW: Create AImageReader (Native)
    media_status_t status = AImageReader_new(
        width,
        height,
        AIMAGE_FORMAT_PRIVATE,  // MediaCodec output format
        3,                      // maxImages (same as Java ImageReader)
        &m_image_reader
    );

    if (status != AMEDIA_OK || !m_image_reader) {
        LogError("Failed to create AImageReader: " + std::to_string(status));
        return false;
    }

    LogInfo("AImageReader created successfully");

    // NEW: Set native image listener (NO JAVA!)
    AImageReader_ImageListener listener{
        .context = this,
        .onImageAvailable = OnImageAvailableStatic
    };

    status = AImageReader_setImageListener(m_image_reader, &listener);
    if (status != AMEDIA_OK) {
        LogError("Failed to set image listener: " + std::to_string(status));
        AImageReader_delete(m_image_reader);
        m_image_reader = nullptr;
        return false;
    }

    LogInfo("Image listener registered (native callback)");

    // NEW: Get ANativeWindow from AImageReader (NO JAVA!)
    status = AImageReader_getWindow(m_image_reader, &m_surface);
    if (status != AMEDIA_OK || !m_surface) {
        LogError("Failed to get window from AImageReader: " + std::to_string(status));
        AImageReader_delete(m_image_reader);
        m_image_reader = nullptr;
        return false;
    }

    LogInfo("ImageReader surface obtained: " +
            std::to_string(reinterpret_cast<uintptr_t>(m_surface)));

    return true;
}
```

### 3.3 Native Callback 구현

**파일:** `MediaCodecSurfaceManager.cpp`

```cpp
// Static callback (called by AImageReader)
void MediaCodecSurfaceManager::OnImageAvailableStatic(void* context, AImageReader* reader) {
    auto* manager = static_cast<MediaCodecSurfaceManager*>(context);
    if (manager) {
        manager->OnImageAvailableCallback(reader);
    }
}

// Instance callback
void MediaCodecSurfaceManager::OnImageAvailableCallback(AImageReader* reader) {
    LogInfo("OnImageAvailableCallback: ENTRY (Native callback)");

    // This callback is invoked on a dedicated AImageReader thread
    // It's safe to call AImageReader_* and AImage_* methods here

    std::lock_guard<std::mutex> lock(m_image_mutex);

    // Acquire latest image (NO JNI!)
    AImage* image = nullptr;
    media_status_t status = AImageReader_acquireLatestImage(reader, &image);

    if (status != AMEDIA_OK || !image) {
        LogWarning("OnImageAvailableCallback: Failed to acquire image: " +
                  std::to_string(status));
        return;
    }

    LogInfo("OnImageAvailableCallback: Image acquired successfully");

    // Get AHardwareBuffer (Direct pointer access - NO JNI!)
    AHardwareBuffer* ahb = nullptr;
    status = AImage_getHardwareBuffer(image, &ahb);

    if (status != AMEDIA_OK || !ahb) {
        LogError("OnImageAvailableCallback: Failed to get AHardwareBuffer: " +
                std::to_string(status));
        AImage_delete(image);
        return;
    }

    // Acquire reference to AHardwareBuffer
    AHardwareBuffer_acquire(ahb);

    // Store current image
    if (m_current_image) {
        AImage_delete(m_current_image);
        m_current_image = nullptr;
    }

    if (m_current_ahardware_buffer) {
        AHardwareBuffer_release(m_current_ahardware_buffer);
        m_current_ahardware_buffer = nullptr;
    }

    m_current_image = image;
    m_current_ahardware_buffer = ahb;

    // Signal waiting thread
    m_image_available = true;
    m_image_cv.notify_one();

    LogInfo("OnImageAvailableCallback: EXIT - Image stored and signaled");
}
```

### 3.4 AcquireLatestImage 수정 (Wait for Callback)

**파일:** `MediaCodecSurfaceManager.cpp`

```cpp
AHardwareBuffer* MediaCodecSurfaceManager::AcquireLatestImage() {
    // REMOVED: JNI calls
    // REMOVED: Java ImageReader.acquireLatestImage()

    // NEW: Wait for OnImageAvailableCallback
    std::unique_lock<std::mutex> lock(m_image_mutex);

    // Wait for image with timeout (100ms)
    bool image_ready = m_image_cv.wait_for(
        lock,
        std::chrono::milliseconds(100),
        [this] { return m_image_available.load(); }
    );

    if (!image_ready || !m_current_ahardware_buffer) {
        LogWarning("AcquireLatestImage: Timeout waiting for image");
        return nullptr;
    }

    // Reset flag
    m_image_available = false;

    LogInfo("AcquireLatestImage: Returning cached AHardwareBuffer");
    return m_current_ahardware_buffer;
}
```

### 3.5 ReleaseImage 수정 (Native)

**파일:** `MediaCodecSurfaceManager.cpp`

```cpp
void MediaCodecSurfaceManager::ReleaseImage() {
    // REMOVED: JNI calls
    // REMOVED: Java Image.close()

    // NEW: Native AImage cleanup
    std::lock_guard<std::mutex> lock(m_image_mutex);

    if (m_current_image) {
        AImage_delete(m_current_image);  // ← Native API
        m_current_image = nullptr;
        LogInfo("AImage deleted (native)");
    }

    if (m_current_ahardware_buffer) {
        AHardwareBuffer_release(m_current_ahardware_buffer);
        m_current_ahardware_buffer = nullptr;
        LogInfo("AHardwareBuffer released");
    }
}
```

### 3.6 Cleanup 수정

**파일:** `MediaCodecSurfaceManager.cpp`

```cpp
void MediaCodecSurfaceManager::Cleanup() {
    // Release current image
    ReleaseImage();

    // REMOVED: JNI cleanup
    // REMOVED: Java ImageReader deletion

    // NEW: Native AImageReader cleanup
    if (m_image_reader) {
        AImageReader_delete(m_image_reader);
        m_image_reader = nullptr;
        LogInfo("AImageReader deleted (native)");
    }

    // Rest of cleanup...
}
```

### 3.7 ProcessAsyncOutputFrame 수정 (Remove Sleep!)

**파일:** `MediaCodecAsyncHandler.cpp`

```cpp
bool MediaCodecAsyncHandler::ProcessAsyncOutputFrame(
    int32_t output_index,
    AMediaCodecBufferInfo* buffer_info,
    VideoFrame& output_frame)
{
    // Phase 1: GPU-synchronized release (unchanged)
    MediaCodecSurfaceManager* surface_manager = m_decoder->GetSurfaceManager();
    if (surface_manager && surface_manager->HasCurrentImage()) {
        void* vk_device = surface_manager->GetVulkanDevice();
        if (vk_device) {
            surface_manager->ReleaseImageAfterGPU(static_cast<VkDevice>(vk_device));
        }
    }

    // Release MediaCodec buffer to ImageReader surface
    media_status_t status = AMediaCodec_releaseOutputBuffer(m_codec, output_index, true);
    if (status != AMEDIA_OK) {
        return false;
    }

    // ====== PHASE 2 CHANGE: Wait for Native Callback ======
    // REMOVED: std::this_thread::sleep_for(std::chrono::milliseconds(5));
    // REMOVED: Retry loop with sleep(2ms)

    // OnImageAvailableCallback will be triggered asynchronously
    // AcquireLatestImage() now waits for callback with condition variable
    // ====== END PHASE 2 CHANGE ======

    // Acquire AHardwareBuffer (callback already happened)
    AHardwareBuffer* ahb = surface_manager->AcquireLatestImage();
    if (!ahb) {
        LogError("ProcessAsyncOutputFrame: Failed to acquire image");
        return false;
    }

    // Rest of code unchanged (Vulkan image creation, etc.)
    // ...

    return true;
}
```

---

## 4. 구현 단계

### Phase 2.1: MediaCodecSurfaceManager 리팩토링 (2-3시간)

**작업 순서:**
1. ✅ MediaCodecSurfaceManager.h - AImageReader 멤버 추가
2. ✅ SetupImageReader() 재구현 (Native API)
3. ✅ OnImageAvailableCallback() 구현
4. ✅ AcquireLatestImage() 수정 (condition variable wait)
5. ✅ ReleaseImage() 수정 (AImage_delete)
6. ✅ Cleanup() 수정 (AImageReader_delete)

**수정 파일:**
- `MediaCodecSurfaceManager.h` - 헤더 수정
- `MediaCodecSurfaceManager.cpp` - 전체 재구현

### Phase 2.2: ProcessAsyncOutputFrame 단순화 (30분)

**작업 순서:**
1. ✅ sleep_for(5ms) 제거
2. ✅ Retry loop 제거
3. ✅ 로직 단순화

**수정 파일:**
- `MediaCodecAsyncHandler.cpp` - ProcessAsyncOutputFrame 수정

### Phase 2.3: 빌드 및 테스트 (1-2시간)

**테스트 항목:**
1. ✅ Android ARM64 빌드 성공
2. ✅ AImageReader 생성 확인 (로그)
3. ✅ Native callback 호출 확인 (로그)
4. ✅ 30 FPS 달성
5. ✅ Sleep 제거 확인 (로그에서 "sleep" 검색)
6. ✅ 레이턴시 측정 (Phase 1 대비)

---

## 5. 제거할 코드

### ❌ 삭제할 Java 파일
- `ImageReaderCallback.java` - 생성했지만 불필요

### ❌ 제거할 JNI 코드
- `MediaCodecSurfaceManager.cpp`:
  - `GetJNIEnv()` 호출 (ImageReader 관련만)
  - `env->FindClass("android/media/ImageReader")`
  - `env->CallObjectMethod()` 호출들
  - `env->NewGlobalRef()` / `DeleteGlobalRef()` (Image 관련)
  - `AHardwareBuffer_fromHardwareBuffer()` 변환

### ❌ 제거할 Workaround
- `std::this_thread::sleep_for(std::chrono::milliseconds(5))`
- `std::this_thread::sleep_for(std::chrono::milliseconds(2))`
- Retry loop (3회 반복)

---

## 6. 성능 예상

### Before (Phase 1 - Java ImageReader)
```
releaseOutputBuffer
  → sleep(5ms)
  → JNI acquireLatestImage (1-5μs)
  → JNI getHardwareBuffer (1-5μs)
  → JNI fromHardwareBuffer (5-10μs)
  → [retry loop: sleep(2ms) x2]

Total: 9-15ms per frame
```

### After (Phase 2 - AImageReader Native)
```
releaseOutputBuffer
  → OnImageAvailableCallback (async, <1ms)
  → AcquireLatestImage (wait on condition_variable)
  → Direct pointer access (<1μs)

Total: 2-5ms per frame
```

**개선:**
- ✅ 레이턴시: 60-70% 감소 (9ms → 3ms)
- ✅ JNI overhead: 100% 제거
- ✅ Java GC pressure: 100% 제거
- ✅ 코드 복잡도: 75% 감소

---

## 7. 검증 체크리스트

### 빌드 검증
- [ ] Android ARM64 빌드 성공
- [ ] AImageReader API 링크 확인 (libmediandk.so)
- [ ] No JNI errors

### 기능 검증
- [ ] AImageReader 생성 성공 (로그)
- [ ] Native callback 호출 확인 (로그: "OnImageAvailableCallback")
- [ ] AImage_getHardwareBuffer 성공
- [ ] Vulkan image 생성 성공

### 성능 검증
- [ ] 30 FPS 이상 달성
- [ ] Sleep 완전 제거 확인 (grep "sleep")
- [ ] 레이턴시 < 5ms (로그 타임스탬프 측정)
- [ ] CPU 사용률 감소

### 안정성 검증
- [ ] 10분 이상 재생 테스트
- [ ] 메모리 누수 확인 (AImage_delete, AHardwareBuffer_release)
- [ ] 크래시 없음 확인

---

## 8. Rollback 계획

Phase 2 구현 중 문제 발생 시:

1. **Git revert:** Phase 1 커밋으로 즉시 복구
2. **Phase 1은 검증됨:** 프로덕션 사용 가능

---

## 9. 다음 단계

### Phase 2 완료 후
1. ✅ 성능 벤치마크 측정 (Phase 1 vs Phase 2)
2. ✅ 문서 업데이트 (COMPLETED_PROJECTS.md)
3. ✅ Phase 1 설계서 보관 (old design reference)

### 향후 최적화 (Optional - Phase 3)
- SyncFence 통합 (API 33+)
- Adaptive buffer size (2-5 frames)
- Frame dropping strategy

---

**문서 상태:** ✅ Ready for Implementation
**예상 구현 시간:** 3-5시간
**우선순위:** 🟢 High (10-20배 성능 향상 예상)
**Java/JNI 의존성:** ❌ None (100% Native C++)