Yes, you'll need a new heatsink and fan for decent temps. Here's how to overclock.
DO NOT DO THIS UNTIL YOU HAVE A NEW HEATSINK/FAN:
This all depends on what kind of motherboard you have. If you can't find a feature, tell us what you see in BIOS.
1. Disable AMD cool and quiet, and also Turbo Core
2. Find the multiplier. Set it to manual. bump it up until it says 3.1ghz to the right.
3. Save and exit BIOS
4. Find a temp guage like HWmonitor or coretemp. Download Prime95 and run it for 30 minutes, watching your temps. If it doesn't crash or stop, you're stable.
5. Repeat until you're at a Ghz you like. Run prime95 for at least 3-5 hours to make sure its stable.
I got to 3.7Ghz on my 960T before having to up my voltage. Just stick to that until you learn more.