Always a good idea to keep it. Just delete the shortcuts and you'll never know it is there until a program (updates, msn, etc) opens it on their own.
If you come across a website that must be opened in Internet Explorer, to open it:
1) Start -> Run...
2) Type iexplore and press ENTER
3) Navigate to the website again.