It depends really. If they look completely off...then I might not get around to playing it. But if I read about a character and find them super interesting/likable I can ignore the art.
For me it's the artstyle and the seiyuus. Even if the story seems "uninteresting" I'll play it if the first two appeal to me. Sometimes I'm also surprised as the story turns out to be fun despite the synopsis.
I think it's up to you. Personally I restart from the beginning because my memory is terrible...so I might not remember certain details that happened during the routes I did complete.