I was in a workshop about privacy recently and at some point, a passionate debate started about Intel's Software Guard Extensions (SGX). Although I have a security background (Master in Information Security), I find it very difficult to understand exactly how SGX works. But I understand that it is an alternative to homomorphic encryption, since it can process data securely way faster than homomorphic encryption does.
At that workshop some people made the argument that there are no guarantees about privacy when it comes to SGX and that, by using SGX, you basically have to consider Intel a trusted third party.
My question is: What are the concerns, or drawbacks, regarding Intel's SGX when it comes to privacy?