When using hardware encoding, may crash or produce videos with pixel misalignment.

[xiaohuirong]

I use the options -c hevc_vaapi -g "<region>" to record windows, and different region selections result in different behaviors for wl-recorder. Running wf-recorder -c hevc_vaapi -g "683,48 1333x140" -f crash.mkv causes the program to crash. wf-recorder -c hevc_vaapi -g "642,317 1322x807" -f misalignment.mkv results in pixel misalignment. However, wf-recorder -c hevc_vaapi -g "637,317 1333x807" -f normal.mkv works correctly. I have tried all supported hardware encoders, and the issue persists. If using software encoding, it works correctly regardless of the selected region.

https://youtu.be/nmSDx5srsY8?si=ELj0hTUGszMeuwRm

• OS: Archlinux 6.4.12-zen1-1-zen
• WM: Wayfire lasted git version
• CPU: i5-13400 / i7-7500U
• GPU: Intel UHD Graphics 730 / Intel HD Graphics 620, both have this issue.

[ammen99]

Does --no-dmabuf help?

[xiaohuirong]

Does --no-dmabuf help?

No effect, same problem.

[xiaohuirong]

My screen's resolution is 2560x1440.

[ammen99]

A quick bisect shows the problem was introduced by #206, with a backtrace from ASAN:

==3972888==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7f5fe09d3000 at pc 0x7f6001449d21 bp 0x7f5ff25fd6f0 sp 0x7f5ff25fcea0
WRITE of size 860160 at 0x7f5fe09d3000 thread T1
    #0 0x7f6001449d20 in __interceptor_memcpy (/lib64/libasan.so.8+0x49d20)
    #1 0x7f5fee919c8f in MosUtilities::MosSecureMemcpy(void*, unsigned long, void const*, unsigned long) (/usr/lib64/dri/iHD_drv_video.so+0x119c8f)
    #2 0x7f5feeac355a in DdiMedia_PutImage(VADriverContext*, unsigned int, unsigned int, int, int, unsigned int, unsigned int, int, int, unsigned int, unsigned int) (/usr/lib64/dri/iHD_drv_video.so+0x2c355a)
    #3 0x7f6001b49cfb in vaPutImage (/lib64/libva.so.2+0xdcfb)
    #4 0x7f600125e941 in vaapi_unmap_frame (/lib64/libavutil.so.57+0x35941)
    #5 0x7f6001257c22 in ff_hwframe_unmap (/lib64/libavutil.so.57+0x2ec22)
    #6 0x7f600124c601 in buffer_replace (/lib64/libavutil.so.57+0x23601)
    #7 0x7f6001257a8b in av_frame_unref (/lib64/libavutil.so.57+0x2ea8b)
    #8 0x7f6001257be9 in av_frame_free (/lib64/libavutil.so.57+0x2ebe9)
    #9 0x7f600126075b in vaapi_transfer_data_to.lto_priv.0 (/lib64/libavutil.so.57+0x3775b)
    #10 0x7f600125827f in av_hwframe_transfer_data (/lib64/libavutil.so.57+0x2f27f)
    #11 0x7f5fff7efa79 in hwupload_filter_frame.lto_priv.0 (/lib64/libavfilter.so.8+0x1efa79)
    #12 0x7f5fff8e3ae8 in ff_filter_graph_run_once.isra.0 (/lib64/libavfilter.so.8+0x2e3ae8)
    #13 0x7f5fff73e76b in get_frame_internal (/lib64/libavfilter.so.8+0x13e76b)
    #14 0x416e90 in FrameWriter::push_frame(AVFrame*, long) ../src/frame-writer.cpp:688
    #15 0x417a08 in FrameWriter::add_frame(unsigned char const*, long, bool) ../src/frame-writer.cpp:743
    #16 0x439806 in write_loop ../src/main.cpp:541
    #17 0x43bd84 in operator() ../src/main.cpp:1191
    #18 0x4402f9 in __invoke_impl<void, main(int, char**)::<lambda()> > /usr/include/c++/12/bits/invoke.h:61
    #19 0x440275 in __invoke<main(int, char**)::<lambda()> > /usr/include/c++/12/bits/invoke.h:96
    #20 0x440143 in _M_invoke<0> /usr/include/c++/12/bits/std_thread.h:258
    #21 0x44008e in operator() /usr/include/c++/12/bits/std_thread.h:265
    #22 0x44004c in _M_run /usr/include/c++/12/bits/std_thread.h:210
    #23 0x7f5fff2dbc02 in execute_native_thread_routine (/lib64/libstdc++.so.6+0xdbc02)
    #24 0x7f5ffe8ae14c in start_thread (/lib64/libc.so.6+0x8b14c)
    #25 0x7f5ffe92f9ff in clone3 (/lib64/libc.so.6+0x10c9ff)

0x7f5fe09d3000 is located 6144 bytes to the left of 699312-byte region [0x7f5fe09d4800,0x7f5fe0a7f3b0)
allocated by thread T1 here:
    #0 0x7f60014ba097 in calloc (/lib64/libasan.so.8+0xba097)
    #1 0x7f5feef36787 in KernelDll_AllocateStates (/usr/lib64/dri/iHD_drv_video.so+0x736787)
    #2 0x7f5feef2adf3 in vp::VpPlatformInterface::InitVPFCKernels(tagKdll_RuleEntry const*, unsigned int const*, unsigned int, unsigned int const*, unsigned int, void (*)(tagKdll_State*)) (/usr/lib64/dri/iHD_drv_video.so+0x72adf3)
    #3 0x7f5feeed5e5e in vp::VpPlatformInterfaceG12Tgllp::InitVpRenderHwCaps() (/usr/lib64/dri/iHD_drv_video.so+0x6d5e5e)
    #4 0x7f5feeee8446 in vp::VpFeatureManagerNext::Init(void*) (/usr/lib64/dri/iHD_drv_video.so+0x6e8446)
    #5 0x7f5feef2cbed in vp::VpPipeline::Init(void*) (/usr/lib64/dri/iHD_drv_video.so+0x72cbed)
    #6 0x7f5feeed92b4 in VpPipelineAdapterLegacy::Init(VpSettings const*, _VP_MHWINTERFACE) (/usr/lib64/dri/iHD_drv_video.so+0x6d92b4)
    #7 0x7f5feeec13df in VpPipelineG12Adapter::Allocate(VpSettings const*) (/usr/lib64/dri/iHD_drv_video.so+0x6c13df)
    #8 0x7f5feef59e2a in DdiVp_CreateContext(VADriverContext*, unsigned int, int, int, int, unsigned int*, int, unsigned int*) [clone .constprop.0] (/usr/lib64/dri/iHD_drv_video.so+0x759e2a)
    #9 0x7f5feeac3472 in DdiMedia_PutImage(VADriverContext*, unsigned int, unsigned int, int, int, unsigned int, unsigned int, int, int, unsigned int, unsigned int) (/usr/lib64/dri/iHD_drv_video.so+0x2c3472)
    #10 0x7f6001b49cfb in vaPutImage (/lib64/libva.so.2+0xdcfb)

Thread T1 created by T0 here:
    #0 0x7f600144b3e6 in __interceptor_pthread_create (/lib64/libasan.so.8+0x4b3e6)
    #1 0x7f5fff2dbcd8 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib64/libstdc++.so.6+0xdbcd8)
    #2 0x43e8af in main ../src/main.cpp:1190
    #3 0x7f5ffe84a50f in __libc_start_call_main (/lib64/libc.so.6+0x2750f)

SUMMARY: AddressSanitizer: heap-buffer-overflow (/lib64/libasan.so.8+0x49d20) in __interceptor_memcpy
Shadow bytes around the buggy address:
  0x0fec7c1325b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fec7c1325c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fec7c1325d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fec7c1325e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fec7c1325f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0fec7c132600:[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fec7c132610: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fec7c132620: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fec7c132630: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fec7c132640: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fec7c132650: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==3972888==ABORTING

This also appears to be related to the geometry, so there is probably a mismatch between the passed sizes somewhere.

[ammen99]

cc @nowrep

[nowrep]

With geometry the dmabuf path isn't taken, so I guess something broke in add_frame?

[nowrep]

I can't reproduce the crash / garbled output with mesa, but I noticed one issue: the sw_format was set incorrectly when using hw encoding without dma-buf capture.

[nowrep]

Please post full log wf-recorder -l ... with current master and at ffcb6f9

From #232 (comment) it looks like the hwfc width/height is somehow wrong?

[ammen99]

I just checked your PR and the crash is gone (I can also reproduce the crash on my 12700, didn't have access to it for some time). @xiaohuirong I hope the crash is fixed for you too?

[xiaohuirong]

I'm very happy the issue has been fixed, thank you all.

[xiaohuirong]

The issue seems to persist when the scaling is set to 1.25 on my 2560x1440 screen.
wf-recorder -g "110,93 800x600" -c hevc_vaapi will result in a chaotic video.