Interesting. Seems to me like latency is not critical in your application and you are doing something more like background processing rather than serving humans on the other side in real time.
replyAlso, some of what you describe sounds expensive. Do you find these techniques + cheaper models better than single-shot with larger models?
Correct, latency is not critical.
replyYes, it can get expensive. We do something in financial markets where the cost of a single end to end task could be $100. That doesn't mean cost doesn't matter though, it matters a lot - if we screw up and it propagates, it's money down the toilet. And if we could get the task down to $20, it would make a difference.