All synthetic data. For this usecase, it was easier because all current generation LLMs, even the small models, are really good at bash commands (and SQL queries too)), so you can reasonably start batches of cheap subagents whose output is reviewed by a more capable model and merge into main training set. After 100k, I had to standing instructions to run the generation loops selectively, meaning only update samples in a given area where we see poor capability.

It is a form of distillation, as long as you're working a very narrow "trivial" topics it works perfectly.

A very small, highly specialized model can use negligible resources (CPU, energy) to accomplish the same task.

For everyday work that happens frequently it's better to have a tiny specialized model instead of making billable API calls or turning your laptop into an 80W space heater for 20 seconds to run a general purpose model.

The large models can be used to generate synthetic training data. Tell them to make up 100,000 tasks paired with the resulting output as a 1-time cost. Then use that to train a small model.

Think of it as distillation, but focused on a specific task.

The models he is using to generate training data are presumably commercial models. He is distilling their bash knowledge into a much smaller model he can run locally fast and cheap.

> That's a really impressive result.

we dont know what the result is and how its impressive.

I appreciate the aside. Interesting observation

I'm literally working on context/harness engineering right now (a set of opencode plugins)

Aside on the aside, I welcome this new era of really personal software. Not Ai's being sycophants, rather being able to easily and quickly change, adapt, or extend software I am not familiar with.

I didn't have a local GPU, so I asked it to go out and find hardware. It found a google TPU v6e which seemed reasonably priced. I gave it my google api key. I told it to use TPU only when training and bring it down afterwards. That's about it.

I am thinking about opensourcing everything, although this is not my main domain or my main startup, so the overhead of huggingface etc seems a bit unnecessary

Edit: will do as soon as possible

Do those statistical models account for declining birth rates or are they based on prior population growth projections?