Unless you activate Teamviewer on your PC, the other party cannot connect. In the remote control mode, you have to give them your ID (which stays the same) and a password that is generated on every program start. If they do not have the password they cannot connect.
A certain amount of trust is involved. If you are sat at the PC being given a tutorial you can see what is going on. You can disconnect at any time. However, if the other party is browsing for a folder or file, they may rummage through Explorer seeing your file and folder listing etc and obviously can see whatever is in or on your desktop. You are exposed in that respect.
Unless you create a VPN, the connection requests go via Teamviewer's server. I suppose that's no worse than using a DNS or ISP but it means someone has tabs on who you are connecting to. In reviews of Teamviewer I've not seen anyone comment on that as yet another company tracking user activity, however benign.