Build Fails at "undefined reference to `XDefaultScreen'" (and Friends)

[Chiitoo]

Operating System Info

Other

Other OS

Gentoo Linux

OBS Studio Version

Git

OBS Studio Version (Other)

No response

OBS Studio Log URL

N/A

OBS Studio Crash Log URL

No response

Expected Behavior

Build is >= successful.

Current Behavior

OBS Studio fails to build unless FFmpeg is linked to 'libX11' like so:

FAILED: plugins/obs-x264/obs-x264-test 
: && /usr/bin/x86_64-pc-linux-gnu-gcc -O2 -march=x86-64 -pipe -pipe -frecord-gcc-switches -fno-diagnostics-color -fmessage-length=0 -Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,--defsym=__gentoo_check_ldflags__=0     -Wl,--dependency-file=plugins/obs-x264/CMakeFiles/obs-x264-test.dir/link.d shared/opts-parser/CMakeFiles/opts-parser.dir/opts-parser.c.o plugins/obs-x264/CMakeFiles/obs-x264-test.dir/obs-x264-test.c.o -o plugins/obs-x264/obs-x264-test  -Wl,-rpath,/var/tmp/portage/media-video/obs-studio-31.0.0/work/obs-studio-31.0.0_build/libobs  libobs/libobs.so.30 && :
/usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../x86_64-pc-linux-gnu/bin/ld: libobs/libobs.so.30: undefined reference to `XDefaultScreen'
/usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../x86_64-pc-linux-gnu/bin/ld: libobs/libobs.so.30: undefined reference to `XLookupString'
/usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../x86_64-pc-linux-gnu/bin/ld: libobs/libobs.so.30: undefined reference to `XOpenDisplay'
/usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../x86_64-pc-linux-gnu/bin/ld: libobs/libobs.so.30: undefined reference to `XCloseDisplay'
collect2: error: ld returned 1 exit status
[209/335] : && /usr/bin/x86_64-pc-linux-gnu-gcc -O2 -march=x86-64 -pipe -pipe -frecord-gcc-switches -fno-diagnostics-color -fmessage-length=0 -Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,--defsym=__gentoo_check_ldflags__=0     -Wl,--dependency-file=plugins/obs-ffmpeg/ffmpeg-mux/CMakeFiles/obs-ffmpeg-mux.dir/link.d plugins/obs-ffmpeg/ffmpeg-mux/CMakeFiles/obs-ffmpeg-mux.dir/ffmpeg-mux.c.o -o plugins/obs-ffmpeg/ffmpeg-mux/obs-ffmpeg-mux  -Wl,-rpath,/var/tmp/portage/media-video/obs-studio-31.0.0/work/obs-studio-31.0.0_build/rundir/RelWithDebInfo/lib64:/var/tmp/portage/media-video/obs-studio-31.0.0/work/obs-studio-31.0.0_build/libobs:  libobs/libobs.so.30  /usr/lib64/libavcodec.so  /usr/lib64/libavutil.so  /usr/lib64/libavformat.so && cd /var/tmp/portage/media-video/obs-studio-31.0.0/work/obs-studio-31.0.0_build/plugins/obs-ffmpeg/ffmpeg-mux && /usr/bin/cmake -E make_directory /var/tmp/portage/media-video/obs-studio-31.0.0/work/obs-studio-31.0.0_build/rundir/RelWithDebInfo/bin && /usr/bin/cmake -E copy_if_different /var/tmp/portage/media-video/obs-studio-31.0.0/work/obs-studio-31.0.0_build/plugins/obs-ffmpeg/ffmpeg-mux/obs-ffmpeg-mux /var/tmp/portage/media-video/obs-studio-31.0.0/work/obs-studio-31.0.0_build/rundir/RelWithDebInfo/bin
FAILED: plugins/obs-ffmpeg/ffmpeg-mux/obs-ffmpeg-mux 
: && /usr/bin/x86_64-pc-linux-gnu-gcc -O2 -march=x86-64 -pipe -pipe -frecord-gcc-switches -fno-diagnostics-color -fmessage-length=0 -Wl,-O1 -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,--defsym=__gentoo_check_ldflags__=0     -Wl,--dependency-file=plugins/obs-ffmpeg/ffmpeg-mux/CMakeFiles/obs-ffmpeg-mux.dir/link.d plugins/obs-ffmpeg/ffmpeg-mux/CMakeFiles/obs-ffmpeg-mux.dir/ffmpeg-mux.c.o -o plugins/obs-ffmpeg/ffmpeg-mux/obs-ffmpeg-mux  -Wl,-rpath,/var/tmp/portage/media-video/obs-studio-31.0.0/work/obs-studio-31.0.0_build/rundir/RelWithDebInfo/lib64:/var/tmp/portage/media-video/obs-studio-31.0.0/work/obs-studio-31.0.0_build/libobs:  libobs/libobs.so.30  /usr/lib64/libavcodec.so  /usr/lib64/libavutil.so  /usr/lib64/libavformat.so && cd /var/tmp/portage/media-video/obs-studio-31.0.0/work/obs-studio-31.0.0_build/plugins/obs-ffmpeg/ffmpeg-mux && /usr/bin/cmake -E make_directory /var/tmp/portage/media-video/obs-studio-31.0.0/work/obs-studio-31.0.0_build/rundir/RelWithDebInfo/bin && /usr/bin/cmake -E copy_if_different /var/tmp/portage/media-video/obs-studio-31.0.0/work/obs-studio-31.0.0_build/plugins/obs-ffmpeg/ffmpeg-mux/obs-ffmpeg-mux /var/tmp/portage/media-video/obs-studio-31.0.0/work/obs-studio-31.0.0_build/rundir/RelWithDebInfo/bin
/usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../x86_64-pc-linux-gnu/bin/ld: libobs/libobs.so.30: undefined reference to `XDefaultScreen'
/usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../x86_64-pc-linux-gnu/bin/ld: libobs/libobs.so.30: undefined reference to `XLookupString'
/usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../x86_64-pc-linux-gnu/bin/ld: libobs/libobs.so.30: undefined reference to `XOpenDisplay'
/usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../x86_64-pc-linux-gnu/bin/ld: libobs/libobs.so.30: undefined reference to `XCloseDisplay'
collect2: error: ld returned 1 exit status

Steps to Reproduce

1. Build FFmpeg without linking to 'libX11'.
2. Build OBS Studio.

Anything else we should know?

Perhaps this is by design, but it seems that after 30.2.3, which I still built using the old CMake path, '/usr/lib64/libobs.so' was linked directly to 'libX11.so.6', like so:

$ lddtree /usr/lib64/libobs.so
libobs.so => /usr/lib64/libobs.so (interpreter => none)
    libavcodec.so.60 => /usr/lib64/libavcodec.so.60
        libvpx.so.9 => /usr/lib64/libvpx.so.9
        libwebpmux.so.3 => /usr/lib64/libwebpmux.so.3
        libdav1d.so.7 => /usr/lib64/libdav1d.so.7
        libaom.so.3 => /usr/lib64/libaom.so.3
        libfdk-aac.so.2 => /usr/lib64/libfdk-aac.so.2
        libmp3lame.so.0 => /usr/lib64/libmp3lame.so.0
        libopenjp2.so.7 => /usr/lib64/libopenjp2.so.7
        libopus.so.0 => /usr/lib64/libopus.so.0
        libtheoraenc.so.1 => /usr/lib64/libtheoraenc.so.1
            libogg.so.0 => /usr/lib64/libogg.so.0
        libtheoradec.so.1 => /usr/lib64/libtheoradec.so.1
        libvorbis.so.0 => /usr/lib64/libvorbis.so.0
        libvorbisenc.so.2 => /usr/lib64/libvorbisenc.so.2
        libwebp.so.7 => /usr/lib64/libwebp.so.7
            libsharpyuv.so.0 => /usr/lib64/libsharpyuv.so.0
        libx264.so.164 => /usr/lib64/libx264.so.164
        libx265.so.209 => /usr/lib64/libx265.so.209
            libstdc++.so.6 => /usr/lib/gcc/x86_64-pc-linux-gnu/14/libstdc++.so.6
                libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/14/libgcc_s.so.1
        libxvidcore.so.4 => /usr/lib64/libxvidcore.so.4
        libopenh264.so.7 => /usr/lib64/libopenh264.so.7
        libva.so.2 => /usr/lib64/libva.so.2
    libavformat.so.60 => /usr/lib64/libavformat.so.60
        libbz2.so.1 => /usr/lib64/libbz2.so.1
        libssl.so.3 => /usr/lib64/libssl.so.3
        libcrypto.so.3 => /usr/lib64/libcrypto.so.3
    libavutil.so.58 => /usr/lib64/libavutil.so.58
        libva-drm.so.2 => /usr/lib64/libva-drm.so.2
            libdrm.so.2 => /usr/lib64/libdrm.so.2
    libswscale.so.7 => /usr/lib64/libswscale.so.7
    libswresample.so.4 => /usr/lib64/libswresample.so.4
    libjansson.so.4 => /usr/lib64/libjansson.so.4
    libz.so.1 => /usr/lib64/libz.so.1
    libX11.so.6 => /usr/lib64/libX11.so.6
    libX11-xcb.so.1 => /usr/lib64/libX11-xcb.so.1
    libxcb.so.1 => /usr/lib64/libxcb.so.1
        libXau.so.6 => /usr/lib64/libXau.so.6
        libXdmcp.so.6 => /usr/lib64/libXdmcp.so.6
    libuuid.so.1 => /usr/lib64/libuuid.so.1
    libm.so.6 => /lib64/libm.so.6
    libgio-2.0.so.0 => /usr/lib64/libgio-2.0.so.0
        libglib-2.0.so.0 => /usr/lib64/libglib-2.0.so.0
            libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0
        libgobject-2.0.so.0 => /usr/lib64/libgobject-2.0.so.0
            libffi.so.8 => /usr/lib64/libffi.so.8
        libgmodule-2.0.so.0 => /usr/lib64/libgmodule-2.0.so.0
        libmount.so.1 => /usr/lib64/libmount.so.1
            libblkid.so.1 => /usr/lib64/libblkid.so.1
    libc.so.6 => /lib64/libc.so.6
    ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2

Switching to the new CMake path with 31.0.0, it seems to only happen via FFmpeg:

$ lddtree /usr/lib64/libobs.so
libobs.so => /usr/lib64/libobs.so (interpreter => none)
    libavcodec.so.60 => /usr/lib64/libavcodec.so.60
        libvpx.so.9 => /usr/lib64/libvpx.so.9
        libwebpmux.so.3 => /usr/lib64/libwebpmux.so.3
        libdav1d.so.7 => /usr/lib64/libdav1d.so.7
        libaom.so.3 => /usr/lib64/libaom.so.3
        libfdk-aac.so.2 => /usr/lib64/libfdk-aac.so.2
        libmp3lame.so.0 => /usr/lib64/libmp3lame.so.0
        libopenjp2.so.7 => /usr/lib64/libopenjp2.so.7
        libopus.so.0 => /usr/lib64/libopus.so.0
        libtheoraenc.so.1 => /usr/lib64/libtheoraenc.so.1
            libogg.so.0 => /usr/lib64/libogg.so.0
        libtheoradec.so.1 => /usr/lib64/libtheoradec.so.1
        libvorbis.so.0 => /usr/lib64/libvorbis.so.0
        libvorbisenc.so.2 => /usr/lib64/libvorbisenc.so.2
        libwebp.so.7 => /usr/lib64/libwebp.so.7
            libsharpyuv.so.0 => /usr/lib64/libsharpyuv.so.0
        libx264.so.164 => /usr/lib64/libx264.so.164
        libx265.so.209 => /usr/lib64/libx265.so.209
            libstdc++.so.6 => /usr/lib/gcc/x86_64-pc-linux-gnu/14/libstdc++.so.6
                libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/14/libgcc_s.so.1
        libxvidcore.so.4 => /usr/lib64/libxvidcore.so.4
        libopenh264.so.7 => /usr/lib64/libopenh264.so.7
        libva.so.2 => /usr/lib64/libva.so.2
    libavformat.so.60 => /usr/lib64/libavformat.so.60
        libbz2.so.1 => /usr/lib64/libbz2.so.1
        libssl.so.3 => /usr/lib64/libssl.so.3
        libcrypto.so.3 => /usr/lib64/libcrypto.so.3
    libavutil.so.58 => /usr/lib64/libavutil.so.58
        libva-drm.so.2 => /usr/lib64/libva-drm.so.2
            libdrm.so.2 => /usr/lib64/libdrm.so.2
        libvdpau.so.1 => /usr/lib64/libvdpau.so.1
        libX11.so.6 => /usr/lib64/libX11.so.6
    libswscale.so.7 => /usr/lib64/libswscale.so.7
    libswresample.so.4 => /usr/lib64/libswresample.so.4
    libjansson.so.4 => /usr/lib64/libjansson.so.4
    libz.so.1 => /usr/lib64/libz.so.1
    libX11-xcb.so.1 => /usr/lib64/libX11-xcb.so.1
    libxcb.so.1 => /usr/lib64/libxcb.so.1
        libXau.so.6 => /usr/lib64/libXau.so.6
        libXdmcp.so.6 => /usr/lib64/libXdmcp.so.6
    libuuid.so.1 => /usr/lib64/libuuid.so.1
    libm.so.6 => /lib64/libm.so.6
    libgio-2.0.so.0 => /usr/lib64/libgio-2.0.so.0
        libglib-2.0.so.0 => /usr/lib64/libglib-2.0.so.0
            libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0
        libgobject-2.0.so.0 => /usr/lib64/libgobject-2.0.so.0
            libffi.so.8 => /usr/lib64/libffi.so.8
        libgmodule-2.0.so.0 => /usr/lib64/libgmodule-2.0.so.0
        libmount.so.1 => /usr/lib64/libmount.so.1
            libblkid.so.1 => /usr/lib64/libblkid.so.1
    libc.so.6 => /lib64/libc.so.6
    ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2

Something like this will allow for FFmpeg to be built without X11 support, but I'm not sure if it would be desirable:

--- libobs/cmake/os-linux.cmake
+++ libobs/cmake/os-linux.cmake
@@ -47,6 +47,7 @@
 target_link_libraries(
   libobs
   PRIVATE
+    X11::X11
     X11::x11-xcb
     xcb::xcb
     LibUUID::LibUUID

See also: https://bugs.gentoo.org/946057

Thank you!