I almost always choose to play/listen with SE. Particularly so for drama CDs, because when it's audio-only, I can't rely on visuals to guide me through the story or set the mood like in games. So sound effects really play a big part in elevating the listening experience, creating/setting atmosphere, and helping me better understand shifts from scene to scene.
The sound effects have to be realistic/natural enough, though. Sometimes the SE in R18 scenes are so bad (I vaguely remember one doujin CD with "thrusting" sounds that just reminded me of a washing machine with a broken drum), or so obnoxiously loud (taking banging too literally, I guess), they break my immersion and make me want to laugh.
So yeah... it's usually SE On/With SE for me. Unless they're terrible, in which case I'd rather have no sound effects at all.