undefined

points

[-]

The secret sauce is training data. They’re not just taking advantage of more compute (which obviously is necessary but as mentions basically a commodity). They are paying billions to data labelers and making judgements about the nature of the training data they best need to make the product they want. This seems to get pushed aside as a minor point but it’s the primary differentiator of the big labs.

by password543212 hours ago|

parent|

[-]

As a I said, compute and data. But LLMs can be distilled, so even their data is not much of a secret sauce.

by reinitctxoffset1 hours ago|

prev|

[-]

I'm pretty sure at this point that Anthropic is training mixture models (at least in the heavy pre-train) and deploying them dense with explicit loss on thinking trace coherence.

Having a thinking trace that is legible, coherent, and immediately implies the explicit turn output and/or tool use seems difficult if not impossible to reliably get from mixture models.

I predict MoE is a transitional technology, it's got too many problems and the benefits are...kinda grandfathered into the dogma at this point.

by charcircuit32 minutes ago|

parent|

[-]

>I predict MoE is a transitional technology

While scaling laws hold (more weights = better), and time / financial costs are not trivial the incentives are in place to have MoE. MoE means you can have more weights without increasing the critical path of evaluating it.

I am curious what you believe the problems with it that would cause people to prefer using less weights. I'm not following what you mean by MoE can't have legible thinkings trace or tool use when existing models with MoE can.

by dominotw1 hours ago|

prev|

[-]

even meta that sucks at doing anything is releasing frontier models. making an top ai is easier than making twitter clone( threads) if you have enough money.

by password543211 hours ago|

parent|

[-]

I mean the problem with Threads was lack of user engagement. The same could possibly still be said about their models.

by dominotw1 hours ago|

parent|

[-]

yes ofcourse. But engagement needs strategy and execution.