I am curious of what is the risk of actually handing over ones fob to valets and car washes.
I think you've indirectly answered your own question, because right after that, you said,
From what I have learned of security, physical access is always a very large risk.
Giving your fob to a valet or car wash attendant is basically giving them unfettered direct physical access to your vehicle - and often, doing this involves them taking your vehicle out of your direct sight for at least a short period of time. Doing so has obvious risks. They have physical access to your vehicle and it's systems. The potential threats range from mild hooliganism (they could do a smoky burnout and wear down your tires) to all kinds of simple to more sophisticated attacks:
- Reading important documents you've left in your car. Many people leave insurance and registration documents in their vehicle, which may have your name, address, and other personal information. This information could be used to impersonate you, steal your identity, or track you down later.
- Stealing easy to remove low value items (pocket change in your center console, your spare tire) or perhaps more difficult to remove, high value items (swapping some valuable part for a fake or a damaged part from another vehicle).
- Planting a tracker or in your vehicle.
- Programming their own fob to work with your vehicle, giving them access to your vehicle in the future.
- Planting malicious software in the vehicle's ECU, entertainment system, or other electronic system.
- and so on...
Of course, it's debatable how likely any of these are, given that many of them are easy to detect (if your pocket change is gone after the car wash, it's pretty obvious someone took it) or would require planning, resources, and sophistication not likely to be applied to a random valet customer.