There is no perfect way to ensure that a program is not malicious, short of extensive reverse engineering. Even with access to the source code, you would need to audit it pretty heavily. While major projects often do get such audits, it is not a panacea. The best you can do is threat reduction.
Does not send any USB data to an external server without my knowledge
This can be mitigated by enabling firewall rules that limit or outright prevent sending traffic over the network for the user that the analyzer is running as. You could also run the analysis on an airgapped computer with no connectivity to the internet.
Does not install any key loggers or other malware like software
While antivirus can detect some simple threats, it is easy to get around it. You could either run from a read-only live DVD, or use Mandatory Access Controls. This technique is similar in purpose to a security-oriented sandbox. MIC is a MAC for Windows, and AppContainer is a sandbox-like isolation feature for Windows. Linux and related operating systems have even more extensive isolation.
Does not screw up my computer. This is what I'm worried about more than anything. For example, I read that another USB debugging tool blocked all USB traffic including mouse/keyboard and rendered the computer useless until the user rolled back their version of Windows.
While you may be able to reduce the chance of software being malicious, knowing how bugs in it may interact with bugs in your operating system is even harder. A potential way to reduce this risk is to use USB packet capturing software and analyze the actual packet dumps using a separate, unprivileged program, allowing you to segment the privileges required to complete your task.
Edit: this is a broader question to related to how one may validate a piece of software when that software's source code is not available. I realise that a certificate helps, but even then, a certificate doesn't guarantee that a piece of software is not doing sneaky things.
There are a few things you can do to reduce the risks:
- Use open source software, ideally popular software that is well-audited.
- Only use signed software, and verify them using programs such as GnuPG.
- If you are particularly paranoid, only use software with reproducible builds.
On Linux, you can use the usbmon kernel driver to capture raw USB traffic, then various packet analysis tools like Wireshark or Virtual USB Analyzer, both of which are open source. Using a well-known live Linux distro such as Knoppix will help reduce the chances that privileged software will cause significant harm to your installation, while preserving your ability to read and write to your disk and analyze captured USB traffic on the system. This gives you a few benefits:
- The software required for USB protocol analysis is open source, making it easier to trust.
- The software runs in a live environment, making it "amnesic", so changes vanish on reboot.
- A Linux environment provides far more tools for debugging and low-level analysis than Windows.
- All the mentioned tools are digitally signed, reducing the risk that they have been modified.