In your user CP, under options make sure you have this checked:
"If you are browsing the forums when you receive a new private message, the system can pop-up a notification box informing you of the new message.
[ ]Show New Private Message Notification Pop-up"
Secondly, in your browser you may have to add SOS.com as an exception in your pop-up blocker settings. I had to add this exception in Firefox to make it work.