Modern GPU drivers (not just limited to Linux) are really, really insecure. The way an application communicates with a GPU is through the Direct Rendering Manager, or DRM interface, using
ioctl(2). These syscalls are used to pass references to complex data structures between the userland and kernel. The DRM interface is available in the form of two character devices:
The master node, which handles privileged operations (e.g. display control). It is exposed as the character device
/dev/dri/cardX. It is often only writable by users in the
video group. This device is traditionally kept open by the X server, which makes itself DRM master. The DRM master node has a considerable attack surface area and uses a complex API.
The render node, which handles generally unprivileged operations (e.g. GPGPU computing and rendering). It is exposed as the character device
/dev/dri/renderDX. This device has a considerable attack surface area, despite only handling unprivileged operations.
Permissions checks for the DRM nodes are done when the device file is being opened, with a few exceptions. Once the file descriptor is present, any application with that descriptor in its set will be able to send arbitrary privileged commands to the graphics subsystem. This means that your isolation ends as soon as your untrusted process has a valid file descriptor to a DRM node. The security of the graphics subsystem depends on both the security of the DRM layer (which has a very complex interface), as well as the hardware-specific interfaces defined by the graphics driver itself. Different drivers define different interfaces, with some being more complex than others.
You said you were using seccomp-bpf. This is good! You will be able to vastly reduce the security issues intrinsic in a complex graphics subsystem. Make sure you are filtering any
ioctl which is called against a DRM node. Whitelist the commands which the
ioctl is permitted to send. Although you cannot whitelist the data structures being passed to the kernel, whitelisting the overall command is often sufficient and will vastly reduce the surface area available for an attacker.
A far from exhaustive list of severe graphics driver bugs:
Intel (i915, etc)
Any driver (bug in the underlying DRM subsystem)
Modern x86 hardware is a scary beast. It is complex and not designed with security in mind. Even if absent of driver software vulnerabilities, GPUs can still pose a threat simply due to their complex nature and interaction with a complicated computer architecture over arcane protocols governed by membership-only thousand-page long specifications. Security is Hard™.