I don't listen to all the voiced lines either. I usually listen to everything only for my favorite characters, and for the rest, once I've read enough of the text, I just skip ahead. If I'm having trouble focusing or a part feels kind of trivial, I'll speed through it and just follow the general flow.
I also really dislike stopping in the middle of a game. If I don't do it this way, I end up sitting there for hours and forcing myself to finish in one go, so this is kind of inevitable for me. That said, when I clear a game quickly like this, I almost always end up wanting to go back and savor my favorite route more carefully, which sometimes means I spend twice as much time overall, lol.
If the goal is purely speed, I think it helps to set the text display speed to the maximum and get used to instantly reading the full text as soon as you skip. I've been practicing that myself, and I've definitely gotten faster over time.