Get a temp program and
Prime95
Go into your BIOS. Look for the multiplier. It's usually at auto or a 2-digit number.
Increase this by .5 or 1, depending on what your BIOS can do.
Save changes, restart.
Once in windows, open the temp program and prime95. Start prime95 and leave it on for about 10-20 minutes. Watch your temperatures.
Repeat until you reach 4Ghz or it fails Prime95. It'll either say one of your cores failed or it will bluescreen. If this happens, return to BIOS and decrease the multiplier by .5 or 1.
Once you get a good overclock and your temps aren't too high, run prime95 for at least 2 hours to make sure there's no problems.