AudienceRestriction is pretty much what you have gathered. It is a validity condition for an assertion. In particular it declares that the assertion's semantics are only valid for the relying party named by URI in that element.
The purpose is to restrict the conditions under which the assertion is valid, and to optionally provide terms and conditions relating to such validity. So the semantics of the element have to do with the scope and conditions of the trust relationships. From SAML 2.0 Core, Section 18.104.22.168(PDF):
Although a SAML relying party that is outside the audiences specified
is capable of drawing conclusions from an assertion, the SAML
asserting party explicitly makes no representation as to accuracy or
trustworthiness to such a party...
<AudienceRestriction> element allows theSAML asserting party to
state explicitly that no warranty is provided to such a party in a
machine- andhuman-readable form. While there can be no guarantee that
a court would uphold such a warrantyexclusion in every circumstance,
the probability of upholding the warranty exclusion is
I.e., it's not a code thing but a human (risk management/warranty/trust) thing. If it's used incorrectly modules tend to throw errors - most SP's expect themselves to be listed in the