You can train a traditional classifier on it quite well.

But the one-shot and few-shot classification performance of decision models really needs in-context learning. The most common way of getting that is by grafting a small LLM onto it, which is certainly possible.

At that point though, why not just use a small LLM with audio capabilities. Gemma E2B is a good model that is already setup for this, and training a decision model LoRA adapter is very easy.

Edit: Apparently people have got decision models working on EmbeddingGemma, eg: https://huggingface.co/mrm8488/embeddinggemma-2-ft-typed-dec.... I'm not seeing it ranked on https://huggingface.co/spaces/multimodalart/jev-decision-ind... so I have questions about how well it does in-context learning.

What would a decision model add to and audio transcriber?

It can already classify decently well, and I tried training it as a decision model, It worked well on domains I trained it for. It was beating Liquid's small decision model, but somehow, it never generalized. Out of the trained domains, it was just above chance.

Isn't EmbeddingGemma2 already a decision model? Am I misunderstanding your question?