Deep Fusion is best, when words and phrase patterns in the domain are known. Deep Fusion means to hint the Whisper decoder about the next possible words using LLM-in-the-loop.
replyCan you go into more detail on this? I’ve been putting together the data pipeline for an ASR fine tune but your approach sounds more interesting.
replybasically he is feeding the same input to multiple models, taking their outputs and dumping it int an LLM to sort out what the reason transcription probably is. expensive but effective.
reply