Yeah i would clean them out. it depends on whether you are going from ati to ati, nvidia to ati, nvidia to nvidia or ati to nvidia:
Ati card to Ati card: download the latest drivers, launch the installer and select uninstall. let it do it's magic, reboot and install the new ones.
Ati card to Nvidia card: Use DriverCleaner Pro (free) to clean out all ati drivers, then download latest drivers and install
Nvidia card to Nvidia card: Same as above.
Nvidia card to Ati card: Same as above.