I think your assessment is correct about the energy losses. I'm just not sure about the fix.
If you want to accelerate a weight quickly in a single swipe, (simplified) gradually reducing the mass along the length of the whip imparts more kenetic energy at the end (the whip crack).
So the arm would no longer be rigid, although you can see in the OPs video it isn't actually fully rigid.
Not saying its impossible to create a successful system like that, in fact it sounds sort of a natural/organic design that potential could be closer to the optimal design, but it would be more difficult to accomplish and probably need more testing and design iterations. So I think OPs strategy is more simple and pragmatic in this case.
You're going to transfer the energy quickly, you'll just break any "arm" that's light enough to accelerate with the energy available.
You could maybe use an 'already broken' arm (rope, like a whip) but then it's not a trebuchet.