I recently upgraded from 2 to 4, and I have noticed no difference.
PS, I have 32 bit so only 3.3GB is being used. However, I would still assume there is not a difference.
However, RAM is so cheap that I would just get 4GB's just in case future stuff does use more than 2GB's.