If that custom ROM is using hardware-backed keystore (Trusted Execution Environment (TEE)) for screen lock authentication and file based encryption (FBE), there's no known way to decrypt
/data partition without screen lock code as TEE is tamper-resistant by design.
In FBE, encryption keys are cryptographically bound to screen lock code and TEE. This ensures that just by knowing screen lock code is not enough to decrypt cloned data. TEE refuses to use its own key for key derivation to decrypt data encryption keys unless user authentication is successful. Tampering with TEE image will hard-brick the device even if bootloader is unlocked.
A threat actor will require you to forcefully unlock the device atleast once. The problem with unlocked bootloader is that they can patch boot image (boot.img, contains kernel) without letting you know of the compromise in order to trick you to unlock it and record your screen lock code.
What you need is custom root of trust which is only supported in Pixel and OnePlus. With custom AVB, you can sign your own images for integrity protection and relock the bootloader. This is as secure as running stock android. However, some apps that rely on SafetyNet attestation to verify device integrity will refuse to run because boot state is
Even if you are using custom AVB, you need to check if custom ROM is enforcing encryption. Some custom ROMs disable
forceencrypt flag in fstab, which disables encryption from first boot of the OS. Your data remains in plain text. At this point, just by deleting lock settings files, a thief can remove screen lock authentication.
You should also check if your custom ROM is enforcing SELinux, otherwise any app can make system calls that otherwise wouldn't have been allowed in SELinux
enforcing. Having root in your custom ROM is not really a problem any more than an unlocked bootloader.
Android architecture has a design flaw which is applicable to stock android also. When a user unlocks screen first time since reboot, unlike iOS, android keeps some of the critical keys in memory even after the user relocks the screen. iOS evicts them 10 seconds after the screen is locked. This opens a security hole because as long as some of the data encryption keys are in memory, they can be extracted by law enforcement and can be used to decrypt some of the data in
/data partition. This requires carefully exposing SoC without disconnecting battery.
Stock android and custom root of trust might be able to stop thieves from owning the device (or data), but when it comes to law enforcement, it depends on their motivation until Google considers eviction of critical data encryption keys from memory when the screen is locked again.