Depends o nwhether the USB connectors within the case are destroyes (ie;frayed).. But just having a damaged USB port will in no way effect your computer overall
If you connect it to the motherboard and try inserting a USB device, it just won't work, or will pop up with an error message that the USB port is overloaded.
I have seen multiple instability issues on computers due to damaged USB ports. I would physically disconnect the offending port prior to connecting the computer to it.