A 3rd party supplier is asking us to install their VPN client onto our
corporate machines to enable access to their environment
Rather than install a piece of software on multiple machines, I would prefer to set up the VPN using a dedicated box in your network (if possible), to grant your users transparent access. It depends on the vendor solution. But your existing router/firewall might even be capable of handling the VPN link. This is likely if were are talking about a common software solution such as OpenVPN or Wireguard. For proprietary solutions the options may be more limited.
If you are going to install software you need to think about maintenance. Updates. Unpatched software is software that can be vulnerable. Just think about it, in recent weeks Pulse Secure has suffered multiple flaws, emergency patches were released but some companies still got hacked. Fast reaction is even more crucial when 0-days are released into the wild.
once the connection is establish can a threat actor from the supplier
use that connection to jump on to our company network?
Possibly. Normally the other party should only have access to a single IP address or a limited range. It is up to you to define the requirement, and enforce it. You can use VLANs for segmentation. At least that's the idea when using hardware for the network interconnection. In fact the other party could be more at risk, since you will be initiating the connection to them but the tunnel goes both ways.
But it's difficult to achieve isolation if you install software straight onto the workstations. I wouldn't take this route personally, and I suspect that the other party is simply proposing you a simple and easy solution that works in a few clicks, but security is an afterthought.