If you plug it into the wall, and flip the switch, your psu wont start up, so yeah, its safe. In order for your psu to be started it needs to be connected to something, like your mother board, or a psu tester (which would be the best way to test them). You can get a tester at a fairly inexpensive price, $10 or so.
you can also jump your psu, by switching around the wires, check a guide for that.