... are we effectively allowing adblock servers to see all our browsing history to process them to remove the ads?
The only servers involved when doing ad-blocking in AdBlockPlus, uBlock and several other products are the servers providing the filter rules. These rules are downloaded locally to the client and used there in the ad-blocking browser extension. This means there is no access to some "adblock server" (whatever this is) to block a specific ad or ads from a specific site.
Of course one might create a proxy which includes such ad-filtering and several firewalls also block ads for security reasons. In this case the proxy/firewall has of course knowledge of the site you currently access. And of course an ad-blocking extension also knows which site you access and thus it might in theory send these URL's to some 3rd party.
But considerng on how much tracking and malvertisement ad-blocking removes you are probably more safe and less tracked when using a widely used ad-blocking extension (preferable with open source code so you could verify the behavior) than to keep surfing with ads and tracking enabled.