And that's where all of our helium actually comes from. Any radioactive decay that emits alpha particles generates helium, since alpha particles are just helium nuclei. When that happens underground, the helium can get trapped. It tends to get trapped in the same places that natural gas gets trapped, so natural gas extraction often encounters helium as well.

Similar to oil and gas (although a completely different mechanism), it takes deep time to accumulate, but can be extracted much, much faster. So although new helium is being generated underground all the time, we can still run out in a practical sense.