In a scenario that allows for preparation, such as giving a speech to a public audience, "asides" are fine. You have advance time to prepare them, determine if they're relevant, and include them if they are.
In more natural conversations, however, improvised asides can be damaging!
For example, if you're in a job interview and you answer a question directly, then spiral into a related story about something that happened to you a few years ago, it could be a sign that you're nervous and looking to fill conversational space!
Instead, focus only on what's immediately relevant.
- Jayson Demers (modified by Amr Badran)