It will be possible to send an image that, once decoded, will require enormous amounts of memory ("PNG bomb"), and it is possible that the graphic routines themselves are vulnerable to malicious content (a carefully crafted image that, when decoded, triggers execution of embedded code). There was such a vulnerability almost ten years ago, and while unlikely, another might pop out.
UPDATE: another did. And another, and this one has a CVSS score of 9.3 -- "There is a total compromise of system integrity. There is a complete loss of system protection, resulting in the entire system being compromised"
HTTP_REFERER tag will allow the owner of the image site to know what pages have been visited, and through the use of tracking cookies, some information disclosure is possible (e.g. the malicious site hosts an image which is shared between sites A, B and C. The owner of the image can see that a given user on site A is the same person as another user of site B, since the image set a cookie when used on site A, and now the same cookie is arriving from a user of site B). Depending on the scenario, this might be undesirable.
By embedding in the image parts of the design of the host site, an attacker might trick the user into believing that the site is hosting some content that's not actually there. This requires the HTML/CSS to be vulnerable to a sudden image size change. For example, a stock exchange site might display, in lieu of a 200x80 banner, a 200x600 banner whose topmost rows are identical to the original banner, and the part below is crafted to simulate the stock exchange site and is "pushed over" a stock ticker, and reproduces the same stock ticker - but with different values. A unwary user might then be tricked into believing stock figures that are totally bogus. If enough users get convinced, this might allow a sort of "pump and dump" scheme.
A variation of this which reportedly often happens is when you link the image without appropriate permission. The website owner, who bears the cost of carrying and supplying the image to your viewers, has the possibility of switching the image with something completely different. This is sometimes done on purpose beforehand, i.e., a "juicy" image is seeded in the appropriate forums (e.g. a mash against some football team?), then is switched with something with opposite content (the same with that team's arch-rivals). For added fun, the trolling webmaster may record the original IP's that downloaded the image, and keep sending them the original picture. So fans of team A will publicize a raspberry against their own team, and not understand why everybody is laughing - every time they look at the image, they see it mocking team B. Which serves as an alert: do not assume the image YOU are seeing is the same YOUR USERS are seeing.