But could a website trick you - e.g. into downloading an executable file that connects to the internet outside of TOR and gives away your ip address?
Technically? Of course it could. It looks like that is pretty much exactly what the FBI did in the Playpen case. If you really care, you have to make sure that there is no way for any of your traffic to not go through Tor.
There was actually a DEF CON talk in 2014 that went into this at some length. That's Zoz - Don't Fuck It Up!, which is available on Youtube. The discussion on Tor starts at 21 minutes, and several parts of it are highly relevant to your question:
- At about 23 minutes 30 seconds, a real-life example of a traffic correlation attack based on local (in that case wireless) network traffic to Tor nodes corresponding to participating in illegal actions. That kind of attack does not require attacking Tor at all.
- At about 24 minutes, the moral of the real-life story being don't fail unsafe with Tor. If you are doing something where you are relying on the anonymization provided by Tor, make certain that if you fail to use Tor then traffic cannot leave your computer or LAN. This includes software other than your web browser.
- At about 27 minutes 50 seconds, the presenter starts discussing Tor deanonymization attacks. Bottom line there: even for NSA and GCHQ, at least in 2012, large-scale Tor deanonymization was not possible, but specific cases of Tor deanonymization were possible (that specific claim is made 28 minutes 30 seconds into the video).
However, the bigger question remains: Why on Earth are you downloading and running untrusted software? That's the attack vector in the case you are describing, and the fact that you are doing that over Tor is largely inconsequential. Don't run untrusted software or open untrusted documents on a trusted system.