upvote
And (from a training material point of view) it makes perfect sense.

I would assume that to a LLM, a binary executable is not much worse than actual source code. And it's much denser, just missing the semantics of human-written identifiers. It's the code's pure logic, condensed into machine language.

And there's tons of it, many vendors supply it "freely" (you don't need license keys to train on a binary) It's often battle-tested code that you wouldn't get hold of otherwise, and compared to ingested source code its unlikely that Codex regurgitates a recognizable part of a Windows 2000 DLL into your C++ source code.

reply
That's not quite right - source expresses the idea relative to a language syntax and grammar, whereas machine code (or assembler) implements the idea against a specific target machine, for a specific set of circumstances.

So as an example, there are a bunch of different ways a for or while loop can be implemented in compiled code, but there are also situations where two quite different source constructs can result in the same machine code sequence.

Machine code certainly won't teach an LLM much about how to write good source code.

reply
Being deterministic, a binary executable is just a compressed version of source code.
reply
Compressed doesn't sound quite right. Source code contains abstractions which are often then expanded out into many variants in an executable (most obviously by function inlining, but at a smaller scope also by things like loop unrolling etc)
reply
Lossy compression.
reply
No, it goes the other way. Source code is a lossy compression of the billions of machine operations it can be compiled into. Abstraction is a way of compressing, both in the mechanical LoC sense, as well as in comprehension.
reply
Many things also expand significantly in binary form. Heavy use of templating, for example.
reply
deleted
reply
deleted
reply
Is plate of spaghetti a compressed version of a recipe?

Executables are often larger than their source code.

A toolchain that creates repeatable binary builds is difficult to create. And a nonrepeatable build is potentially nondeterministic, in the sense that uncontrolled or hidden inputs impact the output.

reply
This is very analogous to vector vs raster graphics. In a way, the vector format is strictly more powerful than the raster format as it can be rendered smoothly at any size. Thus the raster graphic, despite being larger in size, is a lossy distillation of the vector graphic.

Also it _is_ possible to gzip a file and get a larger file, there’s an inflection point based on the distribution of symbols in the source where the same compression algorithm produces a larger file than the original

reply
My hunch is that one of the handicaps Google is dealing with (vs. OpenAI and Anthropics) is that Google is not training on pirated media. They are using YouTube, Google book project scans, and purchased copies of media.
reply
I suspect that Anthropic, being public, and under scrutiny, will get sued into compliance from all over the world. It's not a foregone conclusion that the data they trained on remains fair game over the long term.
reply
Perhaps they are not actively collecting it like Ant/OAI but YouTube and Google Books are full of user-uploaded pirated media.
reply
YouTube has a massive system in place to find and block pirated material. I can't remember the last time I saw a simple video rip on YouTube.

Nowadays you're getting gray area stuff like new AI generated stories based in the Harry Potter universe, which is probably copyright violation but it's not actually pirated video, so Content ID doesn't spot it.

reply
OP is talking about binaries
reply
deleted
reply
There is zero chance models are not trained on pirated media and games, especially most popular ones.

Makes complete sense to study things people actually want to use and consume

reply
At least for Google, they have almost every game worth playing documented in precise frame by frame detail on YouTube. They don't need access to the source code, or compiled binaries to reverse engineer game mechanics. They built vision models perfectly capable of ingesting those game play videos and derive game play understanding from them.

Plus, there's so much open source, and game engine SDKs, freely available, that with a few game engineers at Deep Mind, they can write training systems that connect the dots.

It's definitely a gray area because these models are doing what an indie game dev would do, but at a scale and precision that's basically pirating the game, and getting around the rules of copyright to do it legally.

reply
I agree 100% with sentiment. It is inconceivable to me that models were not trained, either directly, or indirectly through other aggregates, on code and binaries and, of course, media, fetched from torrent or other filesharing sites.
reply
wouldn't that information leak out or somehow found out? too many people already involved. With some, like Seedance it's obvious of course.
reply
I thought the fact they fed copyrighted material into the models has been established.
reply
It has leaked out as part of court cases that Meta has trained LLMs using pirate text torrents.

No reason to belive none of the labs have similarly trained on binaries.

reply
Not saying you're wrong, but LLMs can work in areas they've never been exposed to in languages they've never been exposed to. Training is not needed for something to be possible
reply
They also use synthetic generated data to train on too

If it works and it's verified in nine different ways, throw it in the training pile, and then automate the process.

reply
That is how you end up with model collapse which, I assume, we will see more and more of.
reply
There were fears about this 3-4 years ago but they never manifested. All the labs train heavily on synthetic data today
reply
There is absolutely no way to prove that this claim is true because no model advanced enough to do so at a useful level has any fine grained report of its training data that could be used to verify this.
reply
The only reason at this point that there is no way to prove the claim is true is that we cannot inspect the weights, nor query for copyright violations.

I imagine that someone will eventually figure out how to query black-box LLMs for copyright violations without triggering their guardrails. This is an existential threat for anyone running a black-box LLM with stolen data.

reply
We already know how to and have proved it, before the guardrails got "better" (mostly at hiding it). Early versions of Claude and ChatGPT were fed with a paragraph or two of popular fiction and spat out a half dozen paragraphs more, sometimes verbatim. They're significantly lossy compression algorithms. Not judging just based on that, but it's clear that their creators engaged in far more than the already outrageous amounts of copyright infringement that they've admitted and been so gently slapped on the wrist for.
reply
This is trivially true. Why can they work with a tool or API I’ve written or something internal to a company? They definitely have not trained on my local machine from 2020.
reply
I'm pretty sure companies are not throwing away their training data each time they build a model.
reply
Audio plugins are something that lots of people wanted to try making, but few (until now) could. Now they all can, and they do.

They're about to learn the hard way that the market wasn't profitable even without 10x the number of new entrants.

And that a third of the paying customer base (which was not that large to begin with) gave up on plug-ins anyway to generate finished music tracks from prompts.

reply
> How else do you explain all the avalanche of vibe coded audio plugins and DAWs

By them being nothing special?

> appearing this year?

Vibe coding in general became viable last year, Opus 4.5 is Nov 24, 2025.

reply
Yes, plus there is the fact that there is a HUGE amount of open source audio code out there, many, many DAW’s and so on.

I chalk it up to synth and audio guys being among the more innovative users, personally - its a market of folks who have always wanted to design their own DSP/audio-graphs, and were primed for the opportunity that AI/ML presented to do so.

The same goes for the vibe-coded game communities, and as well all the retro guys.

Folks tend to not realize just how much open source is out there in fields/subjects they’re not really familiar with.

reply
Also the 1 Million token context window helped make it more viable.

I've made a bunch of unreleased vibe-coded VST plugins to replace the hundreds that I've bought over the years - and many of the barriers to development weren't the model. Just having belief that the model was capable of the work, a bit like the 4-minute mile. Also knowing to persevere and iterate with the model, not giving up because the first UI is inevitably so amateurish.

There's also a flywheel happening. If you make permanent notes on the process, your rules for UI design, a library of re-usable components and algorithms, tools so the model can judge its own work... the whole process for the next VST just gets much faster & much easier. Same goes for any software you make with AI too.

reply
so weird how AI boosters act like it hasn't been less than a year, isn't it?
reply
It hasn't been. Since Opus 4.5 "letting Dario take the wheel" became useful for the vast majority of programmers.

As of this fall, you are being professionally irresponsible and SHOULD NOT be working in the field if AI is not integral to your SDLC. At a minimum it should be checking your code for bugs and vulnerabilities.

reply
There are legal issues, ethical issues, moral issues, aesthetic issues, economic issues, financial issues, availability issues, longevity issues, consistency issues, reproducibility issues, sovereignty issues, regulatory issues, licensing issues, terms of service issues, and data retention issues that need to be addressed before AI can even be permitted in my software development life cycle - much less made mandatory on pain of unemployment.
reply
What about security issues? What about the fact that if AI isn't hardening your code under your control, it's hacking it under an adversary's?
reply
Did it come across as dismissive of security issues? I apologise. The security issues are exactly what’s being weighed against those other issues. Both the security issues and the other issues are valid issues and it’s valid to weigh them against each other.

And I sort of understand what you’re trying to say in that second line, something along the lines of “the only thing that stops a bad guy with a gun, is a good guy with a gun”, right? Unfortunately it’s poorly phrased.

If AI isn’t hardening my code, then my code is very vulnerable to AI hacking from an adversary, I’ll grant that. But an absence of my AI does not cause the presence of an adversary’s AI in the way that the binary construction claims. (Compare “if you’re not with us, you’re against us”, which is a threat that says the absence of your fealty does cause the presence of my hostility.)

Likewise, I’ll grant that AI hardening my code does mitigate the damage from AI hacking, but again its presence does not cause the absence of an adversary’s AI hacking in the way your phrasing says it does.

reply
this is what i'm talking about
reply
It should be theoretically possible to use C/etc. source code, assembly output, and compiled output in the training set with equivalence mapping between them. That would make it possible for the LLM to learn disassembly and decompiling to some degree.

You don't need software for that aside from the compiler/linker tools (which are free for C/C++ compilers, rust, etc.). You can then permutate/generate the source code using tools/techniques like fuzzing.

There's also things like Intel's instruction set documentation.

reply
An LLM trained on decompilation would be a game changer, but there's no way they'd allow us to use it.
reply
Maybe you could use a regular decompiler. It makes bad variable and function names. Then get an existing LLM to correct the names.
reply
deleted
reply
It's been a thing for a couple of years now e.g.

https://arxiv.org/html/2403.05286v2

reply
torrented pirated apps may contain all sorts of nasties, it would be amusing if the nastiness gets embedded in the LLMs
reply
In that case it would be movie-plot worthy when the LLM was trained on all those embedded viruses and malware. "Believing" it is intentional and then start embedding it into all software it creates for us.
reply
report shows AI botnet is using your fridge as distributed command and control server

"Hello, we're here to see whether or not your fridge is runni-Hey, no not like that, this is serious!"

reply
AIs are quite good at using existing binary analysis tools; they usually do not just operate on binaries in “prompt space” like this
reply