upvote

  Anthropic spent a lot more time editing their pre-training data with higher-quality material. For example, they used actual books for pre-training, because books are generally run through human editors for higher quality. (this is from the copyright lawsuits)
  [ ... ]
  Meanwhile its obvious that companies like Google would use trash data like email and chat logs or Internet forums for pre-training. Just complete trash data.
Google has been scanning every book they can get their hands on for 20 years or more. They have a bunch of patents on non-destructive book scanning gear. Surely they are using that data to train their models.
reply
won't help if they also include trash internet text in the training set as well.
reply
it's kinda known that the gemini models write better english which goes against the "trash data" remarks.
reply
Surely they are destroying scanned books for training
reply
I don't think it pre-training, because surely these labs all have similar data and Google should be very good at ingesting, cleaning, and filtering data.

I honestly think it just their post-training is fantastic and I think it has something to do with them being very opinionated on how they want the model to act that it basically the best at roleplaying a human. Humans are very good at completely and figuring out things that there aren't "trained" on if given basic set of intelligence. So when it working on these long tasks that might not be directly in it training data it still reason through it very well.

They got made fun of for this in the pass because they do a lot of research on this including hiring a philosopher.

This is just my best guess, because you don't see any other labs talk about how they made the model act and Anthropic is basically obsessed with it and talks about it every model release.

I think they just have better "taste" and that taste is their secret sauce.

reply
Microsoft (MSAI) showed that quality >> quantity with their Phi model a couple of years ago. It was a small model, but trained on very high quality data, and did better than much larger models.
reply
This is incorrect.

Everyone uses books as well as internet data.

Anthropics difference is they have a huge data budget for buying RL environments for real world tasks.

reply
I can't say what precisely, but other big companies are also doing what they can to get human data of high quality, just look for all the companies that popped like mushrooms that do ai reinforcement tasks, micro1, ethos, mercor ecc
reply
Where do you get that information from? Or is it just speculation?
reply
That doesn't really make sense because as good as Opus 5.5 is it writes awful terrible, disgusting slop.
reply
Anthropic doesn't want people to use their models for the chat/persona style use cases OpenAI has spent more time and effort optimizing for.

If you look back at GPT2 and the early days of AI safety/risk this was actually something the kinds of AI researchers that joined Anthropic didn't want - RL on coding->using that accelerate development and to bootstrap capabilities across domains was always the plan, not making it easier for people to spam/fake homework or trap people in problematic human-AI attention/relationship tarpits.

I personally find that Opus 5.5 and Fable 5+ are generally fine to chat with but just very wordy and culturally "LessWrong/essay-guy". Which, if a model is going to have a personality, is close to the one that I want (though it does piss me off when it gets preachy about standards Anthropic clearly doesn't hold themselves against)

I also want to immediately recognize AI generated content on the web or from another person, so I'm very glad it has a distinct voice

reply
Yeah maybe that's the case, and maybe I am more sensitive than other people. But I don't really use AI for that sort of thing but even for technical stuff I still find it's writing style a real turn off. Gemini still has lots of AIisms but it's a lot more tolerable.
reply
If the training set is smaller, doesn't that make sense though? There will be much less variety in its style and so its going to make very recognizable output.
reply
Yeah that could be it. I guess the thing about what makes AI writing so repulsive sometimes is not that it's bad like a human would be bad, but that it does the same things over and over.
reply
It's a much, much better writer than literally every other model.
reply
It's better than Opus 5 but it still has that incredibly punchy contrasty LLM style that worse than most other models
reply
I think by now everyone training trillion parameter models is using almost every book, magazine, research paper, etc they can get.
reply
Purely outside looking in, but yes, I'd argue you are right and higher-quality is a major part of it, both for pre- and post-training. There appears to be a far greater willingness to adopt specifically tailored training sets from third-parties, consistently update and build on them.

From simple frontend work to strategically adding virtual rulers when working with LIDAR scans to extract more accurate data, there are very clear points when Anthropic introduces a new external data set into their mix and while I have seen the same from competitors such as Google, there simply appears to be a greater tendency to stick with these once they have them.

As an example, at release it was quite apparent that Gemini 3 Pro, 3 Flash and Opus 4.5 had a similar, possible even the same frontend-focused training base bought in from a third party. But while Anthropic has had both smaller and larger improvements across each subsequent Opus release, I have not noticed such a consistent improvement with Gemini models over the last year. Maybe Anthropic simply buys updates more often, maybe they create more artificial PRs on top, who knows.

Same with the LIDAR scans, there was a point a few months ago when models started to exhibit a very systematic way of handling visual references in some few leather-crafting tasks, enabling whole new ways of working more efficiently combining existing measurements with images. Step-by-step pixel measurements, compensation for angles and poor captures, etc. Have since seen this in other models, again very much feels like a bought in data set that they are pushing as far as it goes.

Then there is 3D capability with models like Muse Wonder Full. I have not used it myself, if it ain't on API, it doesn't exist, but have seen PRs and do find the model output very parsable, despite it being terrible at any modelling task because these capabilities do not emerge from just training on vast amounts of data, they require high-quality, well tailored, step-by-step reference data of the entire process.

Harness wise, while pure output performance is very consistent across harnesses in my evals (I nowadays use both OpenCode via API and the labs own harness three times each to normalise), Anthropic also appears to be far more willing to use the time between releases to prepare features for the next. Stuff like workflows, proper subagent implementation, advisor, etc. often have found there way into Claude Code either in-between models or were first seen with a model that couldn't fully leverage these. Also feel that explained the good overall launch experience Fable 5 delivered, they simply used the time between announcing Glasswing and launching full access to get the UX/DX polished.

Meanwhile, I am still testing GPT-6.2 Astra and Sol so will refrain from making any more judgement on either till I have all data, but apparently there has been an extensive last minute rewriting of the system prompt to improve performance and "behaviour". Given the gap between GPT-6 Astra and yesterday, along with statements by OpenAI staff alluding that they had this model ready for a while, such things should happen in a far more structured, systematic, well thought out manner, not last second after some negative outside feedback. Heck, from what I know Anthropic, unlike any other lab, does zero early access outside the limited hardening for very exclusive partners like Glasswing. Certainly no early access for regular users, reviewers or the like from what I've heard, so Anthropic does all that prep-work for model releases with less feedback than other labs.

reply