I've been using all 4 in the last few months, to me XP64 is the most responsive, but drivers aren't 100% there for it, although gettin better. Theres also a few games that fail to run properly, or even at all in some cases under the O.S.
XP32 you'll probably already know about, will run all your games at a good speed but lacks any support in DX10.
Vista 64 is apparently slightly faster than 32, as is generally said with most 64/32Bit O.S's, I didn't really see much in my case though, although driver support is there for 98% of drivers games and applications in Both 32 & 64Bit versions of Vista.
You will get some slowdown, an there will be games that perform worse than others, but there are also a few games that have performance increases under Vista. Plus you've got DX10. I personally switched back to XP after having just a bit too much performance drop in the main game I play.
It really depends what you're doing / playing when talking about performance differences although Vista is generally considered, an proven to be slower at the majority of day to day tasks. But its still a great OS to work with, depends how much you care about FPS an a few seconds encoding times etc.
