So yeah it looks like you basically have to disable it entirely right now (though the wording in the options are really ambiguous, so I'm not entirely sure what they might mean).
I had a similar problem like this before, but with emails instead. Whenever someone posted a reply in any thread that I have posted in before, I would automatically get an email notification and my inbox took forever to clear out. '~'
Have you tried unchecking the 'Thread Reply Notification' box under Settings>General settings? Just play around with the settings and you should be able to resolve it afterwards.