how much harm it would do? probably ruin your card. you could look into a aftermarket cooler tough. of ofcourse see if you can tape some foam to the inside of your case's sidepanel so absorb some of the noise. (make sure it doesnt touch any of the electronics tough )
yeah, getting an aftermarket cooler would be my advice as well. Though I do know how annoying it is, it is in the best interest to not remove it for the obvious reasons (destroying your card, maybe more). I'll look around, there may be some coolers that are entirely passive.
{EDIT} http://www.newegg.com/Product/Product.aspx?Item=N82E16835118029
ZALMAN VNF100 High Performance Fanless VGA Cooler - $35
Seems to fit your needs, though for a bit of a price. Passive cooling is difficult for something putting out high consistent temperatures.