Just your computer being plugged into the wall being powered off will probably pull an idle load more than your USB ports combined. I mean, the PSU its self has circuitry at that will have a load on the wall outlit even when the computer isn't on. If you have an USB mouse or keyboard, Bluetooth USB dongle, etc, the load will be negligible as stated. It's really something to not be all that concerned over. If you have a phone plugged into your USB ports and it's in the process of charging, then there will be a load of course.
As to wall warts, yes, they should be unplugged if not in use if you want to save a few cents a month on your electric bill. I don't even bother with most.