The author of the post uses standard terminology like entropy coding and arithmetic coding, and cited a paper "in 2023, Google DeepMind released a paper arguing that language modeling and compression are two views of the same thing" which discusses it further.
This blog post is great. Well explained, and clearly took a lot of effort.
I don't interpret it as them claiming to have to discovered it independently.
It’s a blog post. If it was, say, a peer reviewed paper by Hinton or LeCunn that fails to cite Schmidhuber, that would be reasonable criticism in my opinion. (Spoiler: they fail to cite him)
Either the author knew of prior work that argues the same thing and they ignored it, or they didn't know. And if one writes a 1000+ word article premised on this idea, wouldn't one be presumed to know at least in which century the idea originated from?
Arguably these kind of blog posts should be more subject to such criticisms, because the blog posts purport to "teach" the general public about a concept in an authoritative tone (or at least the author seems to pose as knowledgeable in the subject), while for academic papers, everyone who actually reads the paper knows where the ideas came from anyway and it's mainly an issue of attribution (and maybe about fairly distributing the citation count...)
You're asking why casual comments from amateurs made for fun on the internet shouldn't be held to the same standard as those made by funded career academic experts writing for other experts over months and meant as part of the permanent record of the field?
Personally, I think that's a bit like asking why a friend having you over for dinner isn't always an elegant 7-course meal with wine pairings. I guess you can expect that if you want, but to me it sounds like a child expecting to go to Disney every day: ignoring the economic realities of the situation is a recipe for eternal disappointment.
If the blog is a personal collection of "today I've learned" then different standards apply then "my blog is my research publishing platform".
Both are valid blogs. The fact that it ended on HN front page didn't mean it was targeted towards this audience
This should be expected out of everyone. If you don't respect the reader enough to do this, why should we read your posts?
I think papers should be retracted for not citing prior art, even if you weren't aware of it.
[1] https://news.ycombinator.com/item?id=27440093
[2] https://www.computer.org/csdl/proceedings-article/dcc/2000/0...
Nothing about the concept of "intelligence" is nearly as well defined as the other two, nor as well agreed upon.
Intelligence is a broader concept but definitely involves understanding how a system/envoronment works and making predictions about its unfolding, especially actionable ones that allow you to steer that state towards some goal states.
These are not new.
The con is that you don't get to understand and see any of the history of the ideas or even the ballpark when it was discovered, you attribute it to the blog mentally and you don't know what is how new or old and can't reference it properly when talking to others.
It's wild to me what people here expect out of something they got for free and that was offered as a gift.
Perhaps you could contact the author with your suggestions?
When standing on the shoulders of giants it's hard to tell what is below them.
If you followed the data compression scene in the 80s and early 90s, there were plenty of reinventions of LZ-ish and Huffman-ish algorithms (I also coded my own variant...), and people even tried to patent some of them, so at least for the basics I think it is something that many can discover independently; of course in these times, it's more likely they didn't.
As I say most of these I never publish as I'm mainly using writing as a vehicle for thought, but when I do I'm never sure how to flag them. I don't want (imaginary, lets be honest) readers thinking I'm deluded into thinking I've found something new. I want to come up with a tag I can put on them which adds a pithy disclaimer card at the top or something so I feel more comfortable publishing them.
I don't think it's the case here, but worth noting too that LLM-written blog posts adopt this tone seemingly by default.
Never the least bit of surprise, wonder, doubt, or frustration to get in the way of the steady staccato beat of metaphors, conclusions... and three-item lists.
>> a Developer Educator at ngrok with a passion for nerd-sniping developers.
Maybe more the latter than former...
When you shorten a description in a lossy way, you are deciding a priori that some differences in the object don't matter, and it's not because you understand the object, but because it serves your goal of shortening the description.
You can't just assume smaller is better but it often is. And very often it's more information-dense.
A compression program can compress a book written in a language that the author of the program doesn't understand, on a topic he knows little about.
I think "losslessly even" is the wrong way to think about it. Lossless compression often requires less understanding than high quality lossy compression. If you can do a lossy compression that correctly decides what details are unimportant, that's a good sign of understanding.
This is the crux and reminds me of things like mp3 that exploit the nature of human hearing being limited to a frequency range.
But it's frankly a philosophical question what's real or not. No model is going to capture absolutely everything about the thing it models - at that point, it would be the thing. The best we can hope for is that it captures everything we care about.
And no experiment or metric can tell you if you care about the right things. At best it can tell us if we care about a thing given other things we care about. "No cares in, no cares out".
To make it a little more concrete: you could compress a string from back to front. You could build an LLM to help you do that. If you care about file size, that's almost certainly a bad idea, the forward LLM will be better for that purpose. But are there purposes for which the backward LLM might be better? I think that's not so hard to imagine. Often we wonder about "what came before".
I mean, my interpretation is that the question Turing tried to answer is equivalent to "How can we determine whether machines understand humans/human thought?"
This only works when both systems can talk about pretty much arbitrary things, but if you want a more general method for less complex systems, perhaps having one system simulate another system is sufficient. (Which is also another Turing invention)
At this point, I am unaware of a better definition. I know the Chinese Room argument (and I disagree with it), but I'm not aware whether the proponents of that argument have a better definition of understanding other than "well, the Turing Test isn't enough"...
---
PS: Interestingly the issue of compression is highly relevant regarding the Chinese Room argument -- the essential element in the Chinese Room argument is that the information is not compressed...
No. Teaching is compression. Understanding and application is decompression.
You don't memorize the specific details, you memorize the flow of systems. But you get back to the details to put things into practice.
So was I.
> One decompress it when they can generalize and apply it.
Generalizing is compression. Generalizing and applying are very different things.
> So understanding is compressed, but is not the act of compressing.
Understanding (noun) is compressed. Understanding (verb) is the act of creating understanding (noun) and is compression.
> The larger data it represents never existed.
The larger data is everything you take in about the subject. Experience and instruction.
What is it compressing from?
The author acting as if they discovered this independently had me feel the exact same way. Kinda irritating and almost ... disrespectful? Not sure of the right words to describe it tbh
> The ts_zip utility can compress (and hopefully decompress) text files using a Large Language Model. The compression ratio is much higher than with other compression tools.
It's not only an old idea it's been totally done already.