How to get pixel data out of an IDXGISurface created with GPU access only? - winapi

Question

In broad strokes, what I'm trying to accomplish is capture (part of) the screen and transform the capture into a digital image format. The following steps outline what I believe to be the solution:
Set up a Direct3D11CaptureFramePool and subscribe to its FrameArrived event
Gain access to the pixel data in the FrameArrived event delegate
Pass image data into the Windows Imaging Component to do the encoding
My issue is with step 2: While I can get the captured frame, gaining CPU read access to the surface fails. This my FrameArrived event delegate implementation (full repro below):
void on_frame_arrived(Direct3D11CaptureFramePool const& frame_pool, winrt::Windows::Foundation::IInspectable const&)
{
if (auto const frame = frame_pool.TryGetNextFrame())
{
if (auto const surface = frame.Surface())
{
if (auto const interop = surface.as<::Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess>())
{
com_ptr<IDXGISurface> dxgi_surface { nullptr };
check_hresult(interop->GetInterface(IID_PPV_ARGS(&dxgi_surface)));
DXGI_MAPPED_RECT info = {};
// Fails with `E_INVALIDARG`
check_hresult(dxgi_surface->Map(&info, DXGI_MAP_READ));
}
}
}
}
The Map() call is failing with E_INVALIDARG, and the debug layer offers additional, helpful error diagnostics:
DXGI ERROR: IDXGISurface::Map: This object was not created with CPUAccess flags that allow CPU access. [ MISCELLANEOUS ERROR #42: ]
So, now that I know what's wrong, how do I solve this? Specifically, how do I pull the pixel data out of a surface created with GPU access only?
Following is a full repro. It was originally created using the "Windows Console Application (C++/WinRT)" project template. The only change applied is "Precompiled Header: Use (/Yu)" → "Precompiled Header: Not Using Precompiled Headers", to keep this a single file.
It creates a command line application that expects a window handle as its only argument, in decimal, hex, or octal.
#include <winrt/Windows.Foundation.h>
#include <winrt/Windows.Graphics.Capture.h>
#include <winrt/Windows.Graphics.DirectX.Direct3D11.h>
#include <winrt/Windows.Graphics.DirectX.h>
#include <Windows.Graphics.Capture.Interop.h>
#include <windows.graphics.capture.h>
#include <windows.graphics.directx.direct3d11.interop.h>
#include <Windows.h>
#include <d3d11.h>
#include <dxgi.h>
#include <cstdint>
#include <stdio.h>
#include <string>
using namespace winrt;
using namespace winrt::Windows::Graphics::Capture;
using namespace winrt::Windows::Graphics::DirectX;
using namespace winrt::Windows::Graphics::DirectX::Direct3D11;
void on_frame_arrived(Direct3D11CaptureFramePool const& frame_pool, winrt::Windows::Foundation::IInspectable const&)
{
wprintf(L"Frame arrived.\n");
if (auto const frame = frame_pool.TryGetNextFrame())
{
if (auto const surface = frame.Surface())
{
if (auto const interop = surface.as<::Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess>())
{
com_ptr<IDXGISurface> dxgi_surface { nullptr };
check_hresult(interop->GetInterface(IID_PPV_ARGS(&dxgi_surface)));
DXGI_MAPPED_RECT info = {};
// This is failing with `E_INVALIDARG`
check_hresult(dxgi_surface->Map(&info, DXGI_MAP_READ));
}
}
}
}
int wmain(int argc, wchar_t const* argv[])
{
init_apartment(apartment_type::single_threaded);
// Validate input
if (argc != 2)
{
wprintf(L"Usage: %s <HWND>\n", argv[0]);
return 1;
}
auto const target = reinterpret_cast<HWND>(static_cast<intptr_t>(std::stoi(argv[1], nullptr, 0)));
// Get `GraphicsCaptureItem` for `HWND`
auto interop = get_activation_factory<GraphicsCaptureItem, IGraphicsCaptureItemInterop>();
::ABI::Windows::Graphics::Capture::IGraphicsCaptureItem* capture_item_abi { nullptr };
check_hresult(interop->CreateForWindow(target, IID_PPV_ARGS(&capture_item_abi)));
// Move raw pointer into smart pointer
GraphicsCaptureItem const capture_item { capture_item_abi, take_ownership_from_abi };
// Create D3D device and request the `IDXGIDevice` interface...
com_ptr<ID3D11Device> device = { nullptr };
check_hresult(::D3D11CreateDevice(nullptr, D3D_DRIVER_TYPE_HARDWARE, nullptr,
D3D11_CREATE_DEVICE_BGRA_SUPPORT | D3D11_CREATE_DEVICE_DEBUG, nullptr, 0,
D3D11_SDK_VERSION, device.put(), nullptr, nullptr));
auto dxgi_device = device.as<IDXGIDevice>();
// ... so that we can get an `IDirect3DDevice` (the capture frame pool
// speaks WinRT only)
com_ptr<IInspectable> d3d_device_interop { nullptr };
check_hresult(::CreateDirect3D11DeviceFromDXGIDevice(dxgi_device.get(), d3d_device_interop.put()));
auto d3d_device = d3d_device_interop.as<IDirect3DDevice>();
// Create a capture frame pool and capture session
auto const pool = Direct3D11CaptureFramePool::Create(d3d_device, DirectXPixelFormat::B8G8R8A8UIntNormalized, 1,
capture_item.Size());
auto const session = pool.CreateCaptureSession(capture_item);
[[maybe_unused]] auto const event_guard = pool.FrameArrived(auto_revoke, &on_frame_arrived);
// Start capturing
session.StartCapture();
// Have the system spin up a message loop for us
::MessageBoxW(nullptr, L"Stop capturing", L"Capturing...", MB_OK);
}

Answer 1

You must create a 2D texture that can be accessed by the CPU and copy the source frame into this 2D texture, which you can then Map. For example:
void on_frame_arrived(Direct3D11CaptureFramePool const& frame_pool, winrt::Windows::Foundation::IInspectable const&)
{
wprintf(L"Frame arrived.\n");
if (auto const frame = frame_pool.TryGetNextFrame())
{
if (auto const surface = frame.Surface())
{
if (auto const interop = surface.as<::Windows::Graphics::DirectX::Direct3D11::IDirect3DDxgiInterfaceAccess>())
{
com_ptr<IDXGISurface> surface;
check_hresult(interop->GetInterface(IID_PPV_ARGS(&surface)));
// get surface dimensions
DXGI_SURFACE_DESC desc;
check_hresult(surface->GetDesc(&desc));
// create a CPU-readable texture
// note: for max perf, the texture creation
// should be done once per surface size
// or allocate a big enough texture (like adapter-sized) and copy portions
D3D11_TEXTURE2D_DESC texDesc{};
texDesc.Width = desc.Width;
texDesc.Height = desc.Height;
texDesc.ArraySize = 1;
texDesc.CPUAccessFlags = D3D11_CPU_ACCESS_READ;
texDesc.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
texDesc.MipLevels = 1;
texDesc.SampleDesc.Count = 1;
texDesc.Usage = D3D11_USAGE_STAGING;
com_ptr<ID3D11Device> device;
check_hresult(surface->GetDevice(IID_PPV_ARGS(&device))); // or get the one from D3D11CreateDevice
com_ptr<ID3D11Texture2D> tex;
check_hresult(device->CreateTexture2D(&texDesc, nullptr, tex.put()));
com_ptr<ID3D11Resource> input;
check_hresult(interop->GetInterface(IID_PPV_ARGS(&input)));
com_ptr<ID3D11DeviceContext> dc;
device->GetImmediateContext(dc.put()); // or get the one from D3D11CreateDevice
// copy frame into CPU-readable resource
// this and the Map call can be done at each frame
dc->CopyResource(tex.get(), input.get());
D3D11_MAPPED_SUBRESOURCE map;
check_hresult(dc->Map(tex.get(), 0, D3D11_MAP_READ, 0, &map));
// TODO do something with map
dc->Unmap(tex.get(), 0);
}
}
}
}