During my childhood, my family considered it a 'phase' and presumed I would outgrow cartoons specifically anime.
My friends then, had no clue what 'anime' was because they did not know the word, but they liked it and watched it with me.
Later, during my teens, my family thought I would move to Japan since I was constantly watching anime, reading manga, and practicing Japanese. They told me, if I plan on living there, then I may continue with my 'anime lifestyle'.
Of course, I didn't move.
Luckily, my friends (new group of friends) were more understanding and we would share manga if one person didn't have a certain series, another person would lend it to him/her.
Past my teen years and early twenties, my family did not care about the whole 'living in Japan' thing, but rather... they grew to enjoy it as well. I believe they liked how the voice actors had such quality and emotion in their voices and appreciated how they made the anime more fun to watch.
My current group of friends keep making me cosplay in which I decline or respect my love for anime. Lol.
I have another group of friends who do not really care too much about anime because we are just busy with our lives.
In the end, we shouldn't be too concerned about what other people think because we are the ones in control and we are aware of what we watching and doing. We like what we like no matter the age.