Bluetooth uses encryption and the hacker would need time to crack it last I looked how to do it. I wouldn't worry about it unless your at one loaction for a long time and have a hacker actively trying to crack your Bluetooth connection. Someone else can spoof the MAC address of a Bluetooth device as well.
If you have an Android you can try Bluetooth Firewall.