right click the little volume icon at bottom right of task bar, select playback devices, right click the audio device your headset is plugged into (ie- realtek audio) select it as default playback device.
When you want to swap back to the USB device, repeat the process for that device. Same goes for recording devices, unless a program is set to use a specific recording device, it will use the system default.