This actually happened to my friend not long ago we couldn't get in windows at all to change it, when windows loaded the screen keep saying frequency out of range and it restarted computer.
The way they we fixed it was with a xp disc, try going back to a previous time where the settings werent changed if you have recovery on, if not I dont know a particular way to change the resolution in recovery console but (this is a last resort) you could just reinstall windows.
Also try a larger monitor if you put the resolution or frequency up too high, and if your motherboard has onboard video try switching to that temporaraly.