ened/video-v1
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

WIP

Browse Source
This commit is contained in:
ened
2025-10-14 03:20:42 +09:00
parent 379983233a
commit 2f89643e6b
9 changed files with 1646 additions and 190 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1311
todo29.txt
View File

File diff suppressed because it is too large Load Diff

69
vav2/docs/working/mediacodec-recreation.md Normal file
View File

@@ -0,0 +1,69 @@
# MediaCodec Recreation Issue Analysis
## Why MediaCodec Recreation is Needed
### Problem
MediaCodec is initially configured with `surface=nullptr` (CPU mode), but later we need it to output to ImageReader surface (GPU mode) when Vulkan device is registered.
### Android MediaCodec Limitation
`AMediaCodec_setOutputSurface()` API has limitations:
- Only works for video tunneling mode
- **Does NOT work reliably with async mode callbacks**
- Cannot change surface after codec is started in async mode
### Current Flow
1. `Initialize()` creates MediaCodec with surface=nullptr
2. MediaCodec starts with async callbacks
3. `SetVulkanDevice()` is called later
4. ImageReader surface is created
5. Need to switch MediaCodec output to ImageReader surface
6. **setOutputSurface() doesn't work in async mode → Must recreate codec**
## Current Issue After Recreation
Even after recreation, `getInputBuffer()` returns null for buffer indices from callbacks.
### Root Cause
**Race condition after `AMediaCodec_start()`:**
- Callbacks fire immediately when codec starts
- Buffer indices are enqueued (0-9)
- BUT codec internal buffer allocation isn't complete yet
- When `getInputBuffer()` is called with these indices → returns null
### Evidence from Logs
```
23:32:11.938 - Codec started, callbacks fire (indices 0-9 queued)
23:32:12.142 - DecodeFrameAsync gets index 0
23:32:12.142 - getInputBuffer(index=0) → returns null!
```
## Possible Solutions
### Option 1: Wait after start() then clear queue
After `AMediaCodec_start()`, wait 10-50ms for codec to stabilize, then clear old indices and wait for new callbacks.
### Option 2: Retry logic
If `getInputBuffer()` returns null, put index back in queue and retry with next index.
### Option 3: Don't recreate - use software decode path
Accept that async mode with surface switching doesn't work well, fall back to CPU decoding.
### Option 4: Change initialization order (BEST)
- Call `SetVulkanDevice()` BEFORE `Initialize()`
- But this requires changing application code
## Recommendation
Try **Option 1** first - it's the simplest fix:
```cpp
// After AMediaCodec_start()
std::this_thread::sleep_for(std::chrono::milliseconds(50));
// Clear old buffer indices
{
std::lock_guard<std::mutex> lock(m_async_handler->m_async_mutex);
while (!m_async_handler->m_async_input_buffer_queue.empty()) {
m_async_handler->m_async_input_buffer_queue.pop();
}
}
// Wait for new callbacks with valid buffers
```

77
vav2/platforms/android/applications/vav2player/app/src/main/res/layout/activity_main.xml
View File

@@ -1,50 +1,45 @@
<?xml version="1.0" encoding="utf-8"?> <?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android" <FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto" xmlns:app="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools" xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent" android:layout_width="match_parent"
android:layout_height="match_parent" android:layout_height="match_parent"
android:orientation="vertical"
android:fitsSystemWindows="true" android:fitsSystemWindows="true"
tools:context=".MainActivity"> tools:context=".MainActivity">
<!-- Video Display Area --> <!-- Video Display Area - Full Screen -->
<FrameLayout <com.vavcore.player.VulkanVideoView
android:id="@+id/vulkan_video_view"
android:layout_width="match_parent" android:layout_width="match_parent"
android:layout_height="0dp" android:layout_height="match_parent"
android:layout_weight="1"> android:layout_gravity="center" />
<com.vavcore.player.VulkanVideoView <!-- Video Player Overlay -->
android:id="@+id/vulkan_video_view" <com.vavcore.player.VideoPlayerOverlay
android:layout_width="match_parent" android:id="@+id/video_player_overlay"
android:layout_height="match_parent" android:layout_width="match_parent"
android:layout_gravity="center" /> android:layout_height="match_parent" />
<!-- Video Player Overlay --> <!-- Loading overlay -->
<com.vavcore.player.VideoPlayerOverlay <ProgressBar
android:id="@+id/video_player_overlay" android:id="@+id/loading_indicator"
android:layout_width="match_parent" android:layout_width="wrap_content"
android:layout_height="match_parent" /> android:layout_height="wrap_content"
android:layout_gravity="center"
android:visibility="gone"
style="?android:attr/progressBarStyleLarge"
android:indeterminateTint="@color/primary_color" />
<!-- Loading overlay --> <!-- Status and Performance Info Panel - Overlay with semi-transparent background -->
<ProgressBar
android:id="@+id/loading_indicator"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_gravity="center"
android:visibility="gone"
style="?android:attr/progressBarStyleLarge"
android:indeterminateTint="@color/primary_color" />
</FrameLayout>
<!-- Status and Performance Info Panel -->
<LinearLayout <LinearLayout
android:layout_width="match_parent" android:layout_width="match_parent"
android:layout_height="wrap_content" android:layout_height="wrap_content"
android:layout_gravity="bottom"
android:orientation="vertical" android:orientation="vertical"
android:padding="16dp" android:padding="8dp"
android:background="@color/control_background"> android:background="#80000000"
android:clickable="false"
android:focusable="false">
<!-- Status Text --> <!-- Status Text -->
<TextView <TextView
@@ -53,8 +48,12 @@
android:layout_height="wrap_content" android:layout_height="wrap_content"
android:text="@string/status_ready" android:text="@string/status_ready"
android:textColor="@color/text_primary" android:textColor="@color/text_primary"
android:textSize="14sp" android:textSize="12sp"
android:gravity="center_horizontal" /> android:gravity="center_horizontal"
android:shadowColor="#000000"
android:shadowDx="1"
android:shadowDy="1"
android:shadowRadius="2" />
<!-- Performance Metrics --> <!-- Performance Metrics -->
<TextView <TextView
@@ -63,11 +62,15 @@
android:layout_height="wrap_content" android:layout_height="wrap_content"
android:text="@string/performance_idle" android:text="@string/performance_idle"
android:textColor="@color/text_secondary" android:textColor="@color/text_secondary"
android:textSize="12sp" android:textSize="10sp"
android:layout_marginTop="4dp" android:layout_marginTop="2dp"
android:gravity="center_horizontal" android:gravity="center_horizontal"
android:fontFamily="monospace" /> android:fontFamily="monospace"
android:shadowColor="#000000"
android:shadowDx="1"
android:shadowDy="1"
android:shadowRadius="2" />
</LinearLayout> </LinearLayout>
</LinearLayout> </FrameLayout>

8
vav2/platforms/android/applications/vav2player/app/src/main/res/values/themes.xml
View File

@@ -1,10 +1,14 @@
<?xml version="1.0" encoding="utf-8"?> <?xml version="1.0" encoding="utf-8"?>
<resources> <resources>
<style name="Theme.Vav2Player_Android" parent="android:Theme.Material.Light.NoActionBar" /> <style name="Theme.Vav2Player_Android" parent="android:Theme.Material.NoActionBar" />
<style name="Theme.VavCorePlayer" parent="Theme.AppCompat.Light.NoActionBar"> <style name="Theme.VavCorePlayer" parent="Theme.AppCompat.NoActionBar">
<item name="android:windowFullscreen">true</item> <item name="android:windowFullscreen">true</item>
<item name="android:windowContentOverlay">@null</item> <item name="android:windowContentOverlay">@null</item>
<item name="android:colorBackground">@android:color/black</item>
<item name="android:windowBackground">@android:color/black</item>
<item name="android:textColorPrimary">@android:color/white</item>
<item name="android:textColorSecondary">@android:color/white</item>
</style> </style>
</resources> </resources>

197
vav2/platforms/windows/vavcore/src/Decoder/MediaCodecAV1Decoder.cpp
View File

@@ -11,6 +11,8 @@
// MediaCodec list functionality may need alternative implementation // MediaCodec list functionality may need alternative implementation
#include <cstring> #include <cstring>
#include <cstdlib> #include <cstdlib>
#include <thread>
#include <chrono>
#include <sys/system_properties.h> #include <sys/system_properties.h>
#if __ANDROID_API__ >= 29 #if __ANDROID_API__ >= 29
#include <android/api-level.h> #include <android/api-level.h>
@@ -133,7 +135,7 @@ bool MediaCodecAV1Decoder::SetupVulkanPipeline() {
// Initialization helper: Finalize initialization // Initialization helper: Finalize initialization
bool MediaCodecAV1Decoder::FinalizeInitialization() { bool MediaCodecAV1Decoder::FinalizeInitialization() {
m_initialized = true; m_initialized = true;
if (m_buffer_processor) { if (m_buffer_processor && !IsAsyncModeEnabled()) {
m_buffer_processor->ResetPriming(); m_buffer_processor->ResetPriming();
} }
LogInfo("MediaCodec decoder initialization completed successfully"); LogInfo("MediaCodec decoder initialization completed successfully");
@@ -153,17 +155,38 @@ bool MediaCodecAV1Decoder::Initialize(const VideoMetadata& metadata) {
// Continue anyway - DetectHardwareCapabilities logs details // Continue anyway - DetectHardwareCapabilities logs details
} }
// Step 3: Initialize codec with fallback strategy // Step 3: Vulkan device set? ImageReader is MANDATORY (GPU-only requirement)
if (!InitializeCodecWithFallback()) { if (m_surface_manager->GetVulkanDevice()) {
LogWarning("All hardware AV1 decoders failed, falling back to software (dav1d)"); LogInfo("Vulkan device set - setting up ImageReader (GPU h/w processing required)");
m_hardware_accelerated = false;
return false; // Return false to let factory try next decoder (dav1d) JavaVM* javaVM = GetAndroidJavaVM();
if (!javaVM) {
LogError("Vulkan device set but JavaVM unavailable");
return false; // Hard failure - GPU requirement not met
}
m_surface_manager->SetJavaVM(javaVM);
m_surface_manager->SetVideoDimensions(m_width, m_height);
if (!m_surface_manager->SetupImageReader(m_width, m_height)) {
LogError("Vulkan device set but ImageReader setup failed");
return false; // Hard failure - GPU requirement not met
}
m_surface = m_surface_manager->GetAndroidSurface();
if (!m_surface) {
LogError("Vulkan device set but Surface unavailable");
return false; // Hard failure - GPU requirement not met
}
LogInfo("Vulkan zero-copy pipeline ready");
} }
// Step 4: Setup Vulkan pipeline if Vulkan device is already set // Step 4: Initialize codec with fallback strategy
if (!SetupVulkanPipeline()) { if (!InitializeCodecWithFallback()) {
LogWarning("Vulkan pipeline setup failed - continuing with CPU fallback"); LogError("Hardware AV1 decoder initialization failed");
// Not fatal - continue initialization m_hardware_accelerated = false;
return false;
} }
// Step 5: Finalize initialization // Step 5: Finalize initialization
@@ -543,125 +566,56 @@ JNIEnv* MediaCodecAV1Decoder::GetJNIEnv() const {
} }
bool MediaCodecAV1Decoder::SetVulkanDevice(void* vk_device, void* vk_instance, void* vk_physical_device) { bool MediaCodecAV1Decoder::SetVulkanDevice(void* vk_device, void* vk_instance, void* vk_physical_device) {
if (!m_initialized) { // Pass JavaVM to surface manager before setting Vulkan device
LogError("Cannot set Vulkan device - decoder not initialized"); JavaVM* javaVM = GetAndroidJavaVM();
if (javaVM) {
m_surface_manager->SetJavaVM(javaVM);
LogInfo("JavaVM passed to surface manager in SetVulkanDevice()");
} else {
LogWarning("JavaVM not available in SetVulkanDevice() - ImageReader may fail");
}
// Always store Vulkan device in surface manager (works before or after initialization)
bool result = m_surface_manager->SetVulkanDevice(vk_device, vk_instance, vk_physical_device);
if (result) {
LogInfo("Vulkan device registered with surface manager");
} else {
LogError("Failed to register Vulkan device with surface manager");
return false; return false;
} }
// If decoder is not yet initialized, Initialize() will use the stored Vulkan device
if (!m_initialized) {
LogInfo("Decoder not initialized yet - Vulkan device stored for later use during Initialize()");
return true;
}
// If decoder is already initialized, we need to handle it differently
if (!m_hardware_accelerated) { if (!m_hardware_accelerated) {
LogWarning("Vulkan image requires hardware acceleration"); LogWarning("Vulkan image requires hardware acceleration");
return false; return false;
} }
// CRITICAL: Pass JavaVM to surface manager before setting Vulkan device // Check if ImageReader surface is already configured (early initialization path)
// This is needed for ImageReader initialization on decoder thread ANativeWindow* current_surface = m_surface_manager->GetAndroidSurface();
LogInfo("[SetVulkanDevice] About to call GetAndroidJavaVM()..."); if (current_surface && current_surface == m_surface) {
JavaVM* javaVM = GetAndroidJavaVM(); LogInfo("ImageReader surface already configured during Initialize() - no recreation needed!");
LogInfo("[SetVulkanDevice] GetAndroidJavaVM() returned: " + std::string(javaVM ? "VALID" : "NULL") + " (" + std::to_string(reinterpret_cast<uintptr_t>(javaVM)) + ")"); return true; // Early return - avoid recreation
if (javaVM) {
LogInfo("[SetVulkanDevice] JavaVM is valid, passing to surface manager...");
m_surface_manager->SetJavaVM(javaVM);
LogInfo("JavaVM passed to surface manager in SetVulkanDevice()");
} else {
LogError("[SetVulkanDevice] JavaVM is NULL! ImageReader initialization will fail!");
LogWarning("JavaVM not available in SetVulkanDevice() - ImageReader cannot be initialized");
} }
// Delegate to surface manager // Decoder was initialized without Vulkan, now Vulkan device is being set
bool result = m_surface_manager->SetVulkanDevice(vk_device, vk_instance, vk_physical_device); // This is the problematic late-registration path
if (result) { if (m_width > 0 && m_height > 0) {
LogInfo("Vulkan device set successfully"); LogWarning("Vulkan device set AFTER decoder initialization - ImageReader cannot be added");
LogWarning("MediaCodec must be created with surface from the start");
// CRITICAL FIX: If video dimensions are already set (decoder initialized after Vulkan device), LogWarning("Continuing with CPU fallback - video will work but without zero-copy GPU pipeline");
// we need to setup ImageReader and reconfigure MediaCodec! // Return true because decoder is still functional with CPU path
if (m_width > 0 && m_height > 0) { return true;
LogInfo("Setting up ImageReader → VkImage pipeline after Vulkan device registration");
// Set video dimensions for ImageReader allocation
m_surface_manager->SetVideoDimensions(m_width, m_height);
// Setup ImageReader with video dimensions
if (!m_surface_manager->SetupImageReader(m_width, m_height)) {
LogError("Failed to setup ImageReader after Vulkan device set");
return false;
}
// Get Surface from ImageReader for MediaCodec
m_surface = m_surface_manager->GetAndroidSurface();
if (!m_surface) {
LogError("Failed to get Surface from ImageReader after Vulkan device set");
return false;
}
// CRITICAL: MediaCodec cannot dynamically change surface after being configured with nullptr
// We must stop, reconfigure, and restart MediaCodec with the ImageReader surface
LogInfo("Reconfiguring MediaCodec with ImageReader surface");
// Step 1: Cleanup async mode before stopping MediaCodec
CleanupAsyncMode();
// Step 2: Stop MediaCodec
media_status_t status = AMediaCodec_stop(m_codec);
if (status != AMEDIA_OK) {
LogError("Failed to stop MediaCodec for reconfiguration: " + std::to_string(status));
return false;
}
LogInfo("MediaCodec stopped for reconfiguration");
// Step 3: Reconfigure MediaCodec with ImageReader surface
status = AMediaCodec_configure(
m_codec,
m_format,
m_surface, // ImageReader surface
nullptr, // No crypto
0 // Decoder flag
);
if (status != AMEDIA_OK) {
LogError("Failed to reconfigure MediaCodec with ImageReader surface: " + std::to_string(status));
return false;
}
LogInfo("MediaCodec reconfigured with ImageReader surface");
// Step 4: Setup async callbacks AFTER configure but BEFORE start
LogInfo("Setting up async callbacks for reconfigured MediaCodec");
if (InitializeAsyncMode()) {
if (EnableAsyncMode(true)) {
LogInfo("Async callbacks registered successfully after reconfiguration");
} else {
LogWarning("Failed to activate async mode - continuing with sync mode");
}
} else {
LogWarning("Failed to initialize async mode - continuing with sync mode");
}
// Step 5: Start MediaCodec
status = AMediaCodec_start(m_codec);
if (status != AMEDIA_OK) {
LogError("Failed to restart MediaCodec after reconfiguration: " + std::to_string(status));
return false;
}
LogInfo("MediaCodec restarted successfully with ImageReader output");
// Step 6: Re-prime the decoder after reconfiguration
if (m_buffer_processor) {
m_buffer_processor->ResetPriming();
if (m_buffer_processor->PrimeDecoder()) {
LogInfo("MediaCodec re-primed after reconfiguration");
} else {
LogWarning("MediaCodec priming failed after reconfiguration, but continuing");
}
}
}
} }
return result; return result;
} }
bool MediaCodecAV1Decoder::CreateVulkanImage(void* vk_device, void* vk_instance) {
// Delegate to surface manager
return m_surface_manager->CreateVulkanImage(vk_device, vk_instance);
}
bool MediaCodecAV1Decoder::SetupAHardwareBuffer() { bool MediaCodecAV1Decoder::SetupAHardwareBuffer() {
// Delegate to surface manager // Delegate to surface manager
bool result = m_surface_manager->SetupAHardwareBuffer(); bool result = m_surface_manager->SetupAHardwareBuffer();
@@ -753,11 +707,16 @@ bool MediaCodecAV1Decoder::InitializeMediaCodec() {
// Prime the decoder immediately after initialization (before any decoding starts) // Prime the decoder immediately after initialization (before any decoding starts)
// This ensures no concurrent dequeue issues // This ensures no concurrent dequeue issues
if (!m_buffer_processor->PrimeDecoder()) { // NOTE: Skip priming in async mode - callbacks handle buffer management
LogWarning("MediaCodec priming failed, but continuing initialization"); if (!IsAsyncModeEnabled()) {
// Not a fatal error - decoder can work without priming if (!m_buffer_processor->PrimeDecoder()) {
LogWarning("MediaCodec priming failed, but continuing initialization");
// Not a fatal error - decoder can work without priming
} else {
LogInfo("MediaCodec primed successfully during initialization");
}
} else { } else {
LogInfo("MediaCodec primed successfully during initialization"); LogInfo("Skipping priming - async mode uses callback-based buffer management");
} }
return true; return true;

1
vav2/platforms/windows/vavcore/src/Decoder/MediaCodecAV1Decoder.h
View File

@@ -102,7 +102,6 @@ public:
JNIEnv* GetJNIEnv() const; JNIEnv* GetJNIEnv() const;
// Vulkan image support (public for testing) // Vulkan image support (public for testing)
bool CreateVulkanImage(void* vk_device, void* vk_instance);
bool SetupAHardwareBuffer(); bool SetupAHardwareBuffer();
bool CreateSurfaceFromAHardwareBuffer(AHardwareBuffer* buffer); bool CreateSurfaceFromAHardwareBuffer(AHardwareBuffer* buffer);

115
vav2/platforms/windows/vavcore/src/Decoder/MediaCodecAsyncHandler.cpp
View File

@@ -47,11 +47,14 @@ void MediaCodecAsyncHandler::Cleanup() {
m_decoder = nullptr; m_decoder = nullptr;
m_async_processing_active = false; m_async_processing_active = false;
// Clear async queue // Clear async queues
std::lock_guard<std::mutex> lock(m_async_mutex); std::lock_guard<std::mutex> lock(m_async_mutex);
while (!m_async_output_queue.empty()) { while (!m_async_output_queue.empty()) {
m_async_output_queue.pop(); m_async_output_queue.pop();
} }
while (!m_async_input_buffer_queue.empty()) {
m_async_input_buffer_queue.pop();
}
} }
bool MediaCodecAsyncHandler::SupportsAsyncMode() const { bool MediaCodecAsyncHandler::SupportsAsyncMode() const {
@@ -83,8 +86,14 @@ bool MediaCodecAsyncHandler::InitializeAsyncMode() {
// Setup async callbacks // Setup async callbacks
m_async_callbacks.onInputBufferAvailable = [this](int32_t index) { m_async_callbacks.onInputBufferAvailable = [this](int32_t index) {
// Input buffer available - not used in current implementation // Input buffer available - store in queue for DecodeFrameAsync
// Can be used for async input enqueue in future optimization LogInfo("onInputBufferAvailable callback: index=" + std::to_string(index) +
", MediaCodec=" + std::to_string(reinterpret_cast<uintptr_t>(m_codec)));
std::lock_guard<std::mutex> lock(m_async_mutex);
m_async_input_buffer_queue.push(index);
LogInfo("Input buffer queued: index=" + std::to_string(index) +
", queue size=" + std::to_string(m_async_input_buffer_queue.size()));
m_async_condition.notify_one();
}; };
m_async_callbacks.onOutputBufferAvailable = [this](int32_t index, AMediaCodecBufferInfo* bufferInfo) { m_async_callbacks.onOutputBufferAvailable = [this](int32_t index, AMediaCodecBufferInfo* bufferInfo) {
@@ -140,6 +149,9 @@ bool MediaCodecAsyncHandler::InitializeAsyncMode() {
m_async_mode_enabled = true; m_async_mode_enabled = true;
m_async_processing_active = true; m_async_processing_active = true;
LogInfo("Async mode initialized successfully"); LogInfo("Async mode initialized successfully");
// NOTE: Input buffer callbacks will be triggered automatically after MediaCodec start
// No need to wait here - callbacks are asynchronous
return true; return true;
} }
@@ -150,16 +162,42 @@ void MediaCodecAsyncHandler::CleanupAsyncMode() {
LogInfo("Cleaning up async mode"); LogInfo("Cleaning up async mode");
m_async_processing_active = false; m_async_processing_active = false;
// CRITICAL: Unregister async callbacks from MediaCodec BEFORE clearing state
// This prevents lingering callbacks from firing after codec deletion
if (m_codec) {
LogInfo("Unregistering async callbacks from MediaCodec");
media_status_t status = AMediaCodec_setAsyncNotifyCallback(
m_codec,
{
.onAsyncInputAvailable = nullptr,
.onAsyncOutputAvailable = nullptr,
.onAsyncFormatChanged = nullptr,
.onAsyncError = nullptr
},
nullptr // no userdata
);
if (status != AMEDIA_OK) {
LogWarning("Failed to unregister async callbacks: " + std::to_string(status));
} else {
LogInfo("Async callbacks unregistered successfully");
}
}
m_async_mode_enabled = false; m_async_mode_enabled = false;
// Wake up any waiting threads // Wake up any waiting threads
m_async_condition.notify_all(); m_async_condition.notify_all();
// Clear async queue // Clear async queues
std::lock_guard<std::mutex> lock(m_async_mutex); std::lock_guard<std::mutex> lock(m_async_mutex);
while (!m_async_output_queue.empty()) { while (!m_async_output_queue.empty()) {
m_async_output_queue.pop(); m_async_output_queue.pop();
} }
while (!m_async_input_buffer_queue.empty()) {
m_async_input_buffer_queue.pop();
}
LogInfo("Async mode cleanup complete"); LogInfo("Async mode cleanup complete");
} }
@@ -170,20 +208,56 @@ bool MediaCodecAsyncHandler::DecodeFrameAsync(const uint8_t* packet_data, size_t
return false; return false;
} }
// Enqueue input buffer // Wait for input buffer from async callback
ssize_t input_index = AMediaCodec_dequeueInputBuffer(m_codec, 10000); // 10ms timeout ssize_t input_index = -1;
if (input_index < 0) { {
LogWarning("DecodeFrameAsync: No input buffer available"); std::unique_lock<std::mutex> lock(m_async_mutex);
return false;
// Wait for input buffer with progressive timeout
// First decode might need longer wait for callbacks to start
int timeout_ms = m_async_input_buffer_queue.empty() ? 500 : 100; // 500ms first time, 100ms thereafter
bool buffer_available = m_async_condition.wait_for(
lock,
std::chrono::milliseconds(timeout_ms),
[this] { return !m_async_input_buffer_queue.empty() || !m_async_processing_active; }
);
if (!buffer_available || m_async_input_buffer_queue.empty()) {
LogWarning("DecodeFrameAsync: No input buffer available after " + std::to_string(timeout_ms) + "ms (queue size: " +
std::to_string(m_async_input_buffer_queue.size()) + ")");
return false;
}
// Get input buffer index from queue
input_index = m_async_input_buffer_queue.front();
m_async_input_buffer_queue.pop();
LogInfo("DecodeFrameAsync: Got input buffer index " + std::to_string(input_index) +
" from queue (remaining: " + std::to_string(m_async_input_buffer_queue.size()) + ")");
} }
// Log codec state before attempting getInputBuffer
LogInfo("DecodeFrameAsync: About to call getInputBuffer with index=" + std::to_string(input_index) +
", MediaCodec=" + std::to_string(reinterpret_cast<uintptr_t>(m_codec)) +
", async_mode_enabled=" + std::to_string(m_async_mode_enabled) +
", async_processing_active=" + std::to_string(m_async_processing_active));
size_t buffer_capacity = 0; size_t buffer_capacity = 0;
uint8_t* input_buffer = AMediaCodec_getInputBuffer(m_codec, input_index, &buffer_capacity); uint8_t* input_buffer = AMediaCodec_getInputBuffer(m_codec, input_index, &buffer_capacity);
LogInfo("DecodeFrameAsync: getInputBuffer returned: buffer=" +
std::to_string(reinterpret_cast<uintptr_t>(input_buffer)) +
", capacity=" + std::to_string(buffer_capacity));
if (!input_buffer) { if (!input_buffer) {
LogError("DecodeFrameAsync: Failed to get input buffer"); LogError("DecodeFrameAsync: Failed to get input buffer for index " + std::to_string(input_index) +
", buffer_capacity=" + std::to_string(buffer_capacity));
LogError("DecodeFrameAsync: MediaCodec=" + std::to_string(reinterpret_cast<uintptr_t>(m_codec)));
return false; return false;
} }
LogInfo("DecodeFrameAsync: Got input buffer successfully, capacity=" + std::to_string(buffer_capacity));
if (packet_size > buffer_capacity) { if (packet_size > buffer_capacity) {
LogError("DecodeFrameAsync: Packet size exceeds buffer capacity"); LogError("DecodeFrameAsync: Packet size exceeds buffer capacity");
AMediaCodec_queueInputBuffer(m_codec, input_index, 0, 0, 0, 0); AMediaCodec_queueInputBuffer(m_codec, input_index, 0, 0, 0, 0);
@@ -233,6 +307,16 @@ bool MediaCodecAsyncHandler::WaitForAsyncFrame(VideoFrame& output_frame, int tim
return true; return true;
} }
void MediaCodecAsyncHandler::ClearInputBufferQueue() {
std::lock_guard<std::mutex> lock(m_async_mutex);
int cleared_count = 0;
while (!m_async_input_buffer_queue.empty()) {
m_async_input_buffer_queue.pop();
cleared_count++;
}
LogInfo("Cleared " + std::to_string(cleared_count) + " stale input buffer indices from queue");
}
bool MediaCodecAsyncHandler::ProcessAsyncOutputFrame(int32_t output_index, AMediaCodecBufferInfo* buffer_info, VideoFrame& output_frame) { bool MediaCodecAsyncHandler::ProcessAsyncOutputFrame(int32_t output_index, AMediaCodecBufferInfo* buffer_info, VideoFrame& output_frame) {
if (!m_codec || output_index < 0 || !buffer_info) { if (!m_codec || output_index < 0 || !buffer_info) {
LogError("ProcessAsyncOutputFrame: Invalid parameters"); LogError("ProcessAsyncOutputFrame: Invalid parameters");
@@ -329,8 +413,15 @@ bool MediaCodecAsyncHandler::ProcessAsyncOutputFrame(int32_t output_index, AMedi
void MediaCodecAsyncHandler::OnAsyncInputAvailable(AMediaCodec* codec, void* userdata, int32_t index) { void MediaCodecAsyncHandler::OnAsyncInputAvailable(AMediaCodec* codec, void* userdata, int32_t index) {
auto* handler = static_cast<MediaCodecAsyncHandler*>(userdata); auto* handler = static_cast<MediaCodecAsyncHandler*>(userdata);
if (handler && handler->m_async_callbacks.onInputBufferAvailable) { if (handler) {
handler->m_async_callbacks.onInputBufferAvailable(index); // Log codec pointer from callback parameter vs stored pointer
handler->LogInfo("OnAsyncInputAvailable: index=" + std::to_string(index) +
", callback_codec=" + std::to_string(reinterpret_cast<uintptr_t>(codec)) +
", stored_codec=" + std::to_string(reinterpret_cast<uintptr_t>(handler->m_codec)));
if (handler->m_async_callbacks.onInputBufferAvailable) {
handler->m_async_callbacks.onInputBufferAvailable(index);
}
} }
} }

6
vav2/platforms/windows/vavcore/src/Decoder/MediaCodecAsyncHandler.h
View File

@@ -65,6 +65,9 @@ public:
bool DecodeFrameAsync(const uint8_t* packet_data, size_t packet_size, VideoFrame& output_frame); bool DecodeFrameAsync(const uint8_t* packet_data, size_t packet_size, VideoFrame& output_frame);
bool WaitForAsyncFrame(VideoFrame& output_frame, int timeout_ms = 100); bool WaitForAsyncFrame(VideoFrame& output_frame, int timeout_ms = 100);
// Queue management
void ClearInputBufferQueue();
// Async callback handlers (static methods for C callback compatibility) // Async callback handlers (static methods for C callback compatibility)
static void OnAsyncInputAvailable(AMediaCodec* codec, void* userdata, int32_t index); static void OnAsyncInputAvailable(AMediaCodec* codec, void* userdata, int32_t index);
static void OnAsyncOutputAvailable(AMediaCodec* codec, void* userdata, int32_t index, AMediaCodecBufferInfo* bufferInfo); static void OnAsyncOutputAvailable(AMediaCodec* codec, void* userdata, int32_t index, AMediaCodecBufferInfo* bufferInfo);
@@ -100,6 +103,9 @@ private:
// Async output queue // Async output queue
std::queue<AsyncFrameData> m_async_output_queue; std::queue<AsyncFrameData> m_async_output_queue;
// Async input buffer index queue
std::queue<int32_t> m_async_input_buffer_queue;
// Async callbacks // Async callbacks
MediaCodecAsyncCallbacks m_async_callbacks; MediaCodecAsyncCallbacks m_async_callbacks;
}; };

52
vav2/platforms/windows/vavcore/src/VavCore.cpp
View File

@@ -397,25 +397,26 @@ VAVCORE_API VavCoreResult vavcore_open_file(VavCorePlayer* player, const char* f
player->impl->pendingD3DSurfaceType = VAVCORE_SURFACE_CPU; player->impl->pendingD3DSurfaceType = VAVCORE_SURFACE_CPU;
} }
LOGF_DEBUG("[VavCore] Initializing decoder...");
// Initialize decoder
if (!player->impl->decoder->Initialize(player->impl->metadata)) {
LOGF_ERROR("[VavCore] Decoder initialization failed (unsupported format or hardware unavailable)");
player->impl->decoder.reset();
player->impl->fileReader->CloseFile();
return VAVCORE_ERROR_DECODER_UNAVAILABLE;
}
LOGF_DEBUG("[VavCore] Decoder initialized successfully!");
#ifdef ANDROID #ifdef ANDROID
// Apply pending Vulkan device AFTER decoder initialization // CRITICAL: Apply Vulkan device BEFORE decoder initialization
// This allows MediaCodec to be created with ImageReader surface from the start
if (player->impl->has_vulkan_device) { if (player->impl->has_vulkan_device) {
LOGF_DEBUG("[VavCore] Applying pending Vulkan device after decoder initialization..."); LOGF_DEBUG("[VavCore] Applying pending Vulkan device BEFORE decoder initialization...");
LOGF_DEBUG("[VavCore] Vulkan device: %p, instance: %p, physical device: %p", LOGF_DEBUG("[VavCore] Vulkan device: %p, instance: %p, physical device: %p",
player->impl->vulkan_device, player->impl->vulkan_instance, player->impl->vulkan_physical_device); player->impl->vulkan_device, player->impl->vulkan_instance, player->impl->vulkan_physical_device);
// Pre-check: Vulkan device requires JavaVM for ImageReader initialization
// If JavaVM is not available, decoder initialization will 100% fail
JavaVM* javaVM = VavCore::GetAndroidJavaVM();
if (!javaVM) {
LOGF_ERROR("[VavCore] CRITICAL: Vulkan device set but JavaVM unavailable!");
LOGF_ERROR("[VavCore] This means libVavCore.so was not properly loaded or JNI_OnLoad failed.");
LOGF_ERROR("[VavCore] GPU hardware processing requires JNI (Android requirement).");
player->impl->decoder.reset();
player->impl->fileReader->CloseFile();
return VAVCORE_ERROR_INIT_FAILED;
}
bool vulkan_success = player->impl->decoder->SetVulkanDevice( bool vulkan_success = player->impl->decoder->SetVulkanDevice(
player->impl->vulkan_device, player->impl->vulkan_device,
player->impl->vulkan_instance, player->impl->vulkan_instance,
@@ -423,16 +424,29 @@ VAVCORE_API VavCoreResult vavcore_open_file(VavCorePlayer* player, const char* f
); );
if (vulkan_success) { if (vulkan_success) {
LOGF_INFO("[VavCore] Vulkan device successfully registered with decoder"); LOGF_INFO("[VavCore] Vulkan device registered with decoder BEFORE initialization");
} else { } else {
LOGF_WARNING("[VavCore] Failed to register Vulkan device with decoder (will use CPU fallback)"); LOGF_ERROR("[VavCore] Failed to register Vulkan device with decoder");
LOGF_ERROR("[VavCore] GPU hardware processing requirement not met");
player->impl->decoder.reset();
player->impl->fileReader->CloseFile();
return VAVCORE_ERROR_INIT_FAILED;
} }
// Note: We keep has_vulkan_device=true even if registration failed
// This allows retry on next decoder recreation
} }
#endif #endif
LOGF_DEBUG("[VavCore] Initializing decoder...");
// Initialize decoder (now with Vulkan device already set!)
if (!player->impl->decoder->Initialize(player->impl->metadata)) {
LOGF_ERROR("[VavCore] Decoder initialization failed (unsupported format or hardware unavailable)");
player->impl->decoder.reset();
player->impl->fileReader->CloseFile();
return VAVCORE_ERROR_DECODER_UNAVAILABLE;
}
LOGF_DEBUG("[VavCore] Decoder initialized successfully!");
// Apply debug options to newly created decoder // Apply debug options to newly created decoder
player->impl->decoder->SetDebugOptions(&player->impl->debugOptions); player->impl->decoder->SetDebugOptions(&player->impl->debugOptions);
LOGF_DEBUG("[VavCore] Debug options applied to decoder"); LOGF_DEBUG("[VavCore] Debug options applied to decoder");