The pop up is for Private messages, PM's messages that are posted to the forum will get an email notification if you tick the box for one.
When you visit the forum do you see orange ticks down the left hand side? They are the new posts. Try sticking to a few boards at first so that you get used to checking them and how they work first. If you leave the forum window the orange ticks go, so only new posts since your last visit will have them next time you visit and that way it is easy to miss some.
You will get used to it, I know the layout is a bit complicated and its something that we are all working on, but sometimes we forget that newer members find it hard. see how you get on

Yell again if you need help.