You probably think your smartphone is listening to you when it isn't. Those weirdly accurate ad placements often come from other creepy means, such as noticing when you pause ever so slightly while scrolling a web page.
Getting personal information from you via your voice is VERY inefficient because sound files take up a lot of space. It's expensive to store and analyze all those bytes.
You should still be worried about other means of violating your data privacy.