For me, sometimes I don't even make it through the first route if the writing isn't good enough. If the story feels shallow, the plot doesn't have enough substance, or the characters just aren't interesting or compelling to me, I can struggle to even get through the common route.
I think visual novels aren't that different from novels in this regard. They have to capture the reader's attention within the first few chapters, otherwise it's hard to stay invested.
There have also been some games where I forced myself to finish the first route, but still couldn't find the appeal of the game. In those cases, I usually assume that the writing and overall storytelling style of the other routes will probably be similar, so I don't feel very motivated to continue.
That said, there are exceptions, especially games where each route has a completely different tone or feels like an entirely different story. For those, I'd probably judge each route individually rather than letting the first route determine my overall opinion.