Oftentimes I come across an answer that reads something like
On a related note, [...], or _Yes and here's something else [...]
Just an enthusiast got enthusiastic, just like we all should do.
I assume you are talking about 'Answers' that only contain such sidenotes/remarks but don't even try to tackle the question asked?
Sometimes I comment and I put something like
Yes, but how does it answer the question?
and then I flag it as "not an answer".
Exacly like I do as well. This stuff might be realy interesting, but unless the author doesn't try to answer, at least in part, it's not what should be there. The distinguishing feature of C is being a Q&A site focused on concrete and related answers, not user stories or beautiful memories (*1)
Some of my "not an answer" flags were marked as helpful, some weren't, and some of the answers are still there.
Usually it's a community decision, and some may see this point less strict as you and me. Some actually like such stories - especially when about some of the favourites. Being a community, we may have to live with it - still, I will flag it.
So I wonder if we need to have a think about how this community needs to treat these answers. If we're a question-and-answer site, and need to vote these down more aggressively or remove them, as would happen on our sister site stackoverflow, or if we're relaxed and more interested in these interesting tidbits.
I'm well in favour for a rather strict handling of the topic and answer handling as you might know. And to be honest, me flagging such 'answers' down won't change even if there is a more relaxed aproach adopted, as I feel them to be worth a downvote.
Now, when the answer in question also includes a real andswer, just to be followed by lenthy stories, we need to judge it more diferentiated.
For one, if the story is just additional and there is a also a related answer, it comes down to the quality of the answer. I'd be rather relaxed if this is the case - even more so if the answer realy tackles the question before going astray.
Second, there may still be the case that such a story is related to the question. Maybe jsut anecdotal. In this cas I may restain frommarking them down, but this is realy a case by case decision.
*1 - I admit, I'm guilty of this - but I try to put my nostalgia always second to answering the question.