David SmithJan 15, 2023

I'm working on reviving my old podcast searching system using OpenAI's Whisper engine (https://github.com/openai/whisper).

The results so far are amazing. I can run the transcription right on my Mac at roughly 5X realtime, and the accuracy is super impressive. It even gets brand names and weird words right nearly every time.

For example, this segment from The Talk Show where @marcoarment and @gruber argue about how to pronounce databases was perfectly transcribed, down the even the mispronunciations. 🤯

GitHub - openai/whisper: Robust Speech Recognition via Large-Scale Weak Supervision

Robust Speech Recognition via Large-Scale Weak Supervision - openai/whisper

GitHub
Show threadJohn SiracusaJan 15, 2023
@_Davidsmith Can it do speaker identification?
Show threadDavid SmithJan 15, 2023
@siracusa not directly, there are other tools you can run that will segment by speaker, so if you wanted to I suppose you could combine them.
Show threadJohn SiracusaJan 15, 2023
@_Davidsmith Do you know of any that run on the Mac? I’d love transcripts and search for all my podcasts, but I think speaker identification is essential.
Show threadMarcos Huerta

@siracusa @_Davidsmith OpenAI has said their model could do it, but I haven't seen anyone try to use them that way (yet.)

https://github.com/openai/whisper/blob/main/model-card.md

"They may exhibit additional capabilities, particularly if fine-tuned on certain tasks like voice activity detection, speaker classification, or speaker diarization but have not been robustly evaluated in these areas. "

EDITED to link to this discussion and some efforts to combine with pyannote https://github.com/openai/whisper/discussions/264

whisper/model-card.md at main · openai/whisper

Robust Speech Recognition via Large-Scale Weak Supervision - whisper/model-card.md at main · openai/whisper

GitHub
Jan 15, 2023 at 3:40pmWeb