That wouldn't work. The x server talks to your cpu, gets the data off the ram that it needs to render then back to your cpu to get "packaged" to be sent through your bus to you northbridge then to your GPU to get rendered. On top of that there are a ton of physics that needs to be processed. I dont exactly want to get into everything that needs to get processed by your cpu, but that's the most important aspects of it.
In short, no, it will not work. Your cpu will bottleneck. That and the bus is way too slow to process that much data.