On aftermarket coolers, you usually have to replace the stock cpu heatsink mount with the one that comes with the new heatsink.
A new case would help. However, it looks like you have a btx motherboard in there and only a btx case would work. What is the make and model number of pc you have?