I think depends on how is that implemented.
Normally I would say I like to just stop reading for a bit and change the input, but if is not fitting good inside the narrative I prefer not.
I you think about it also video games with strong narrative can enter in this category (Dragon Age, Baldurs Gate, etc...). Having just to puzzle without reason is not my cup of tea, different if I have to do it for entering somewhere, for riddles, I don't know. If also how I do it change whats happens maybe. As a choice.
Preparing recipes for example its ok If I'm a cook.
Also kind of managerial stats game play is ok. Because is growing stats in different way.
But if the story is good, you don't need mini games. At contrary if you add mini games and game play doesn't mean you really need it or that the text you have is good.
So... controversial.