If I'd get a buck every time someone said something like this to me when I was in the 13-18 range, I wouldn't have a ton of money, but it's so very annoying when people tell you this.
Regardless if they're "gifted" or not, regardless if you believe in myths like that or not, let children explore what they want to explore, even if you don't understand what it is or why they want to explore that, just let people explore, regardless of age.
It was such a terrible experience being a young kid growing up, with so many adults spending hours trying to convince me to stop sitting in front of the computer so much doing whatever; "why are you even trying to learn that stuff, you have to go to school to understand anything of this" and so much other similar trash.
Sorry, not your fault and I'm borderline trauma-dumping now, but really sad to see this sort of gatekeeping on HN of all places, age is irrelevant to learning ANYTHING, in my humble opinion at least.
Kids, find anything interesting? Jump into it, ignore what adults tell you, and do whatever you feel like, you'll find your place eventually.
There still will be varyy small number of outliers among youngsters who'd be able to extract tremensous value from such an excercise, but for most that'd be _IMO_ waste of of time, with illusion of understanding w/o actually having any.
Same! I just happened to disagree with your opinion, and frankly, I'd say trying to gatekeep what people learn is closer to "borderline irresponsible" compared to asking people to build/learn/do X.
> youngsters who'd be able to extract tremensous value from such an excercise
But they're youngsters, who are about "extracting value"? Life is about fun, not extraction, not value, not avoiding waste of time but literally enjoy what you do, nothing is more important (IMO).
Then who knows, doing fun stuff sometimes lead to useful stuff, like in my life. But if you only think about "extracting most value for time spent" or similar "optimization strategies", then you'd never discover this part of life.
This is, pardon, demagoguery. There is always "future fun" and "present fun" which a normal person would assign different nonzero weights (https://en.wikipedia.org/wiki/Discounted_utility). Besides, building a LLM _truly_ from the scratch, just using the famous 2017 paper and numpy manuals is not fun at all, esp. for a high schooler.
To you it isn't, is my entire point here. But why extrapolate what you think is fun, to others? Sure, I don't find that fun either (although useful), but who am I to say it isn't fun for others?
Yeah, if you see it as pointless, then I prefer not to. Have a enjoyable day!
This also meant that by the time I was actually offered to take a programming class in school (junior year of HS), I had already been able to self-teach myself well beyond what that class was covering, thanks to just working on random projects that scratched an itch I had at the time, looking up anything I didn't know or understand, and internalizing those concepts over time.
In short though, I definitely agree, young kids and teens (and also, frankly, adults too!) should be encouraged to explore things that they have a passion for, without being told 'you need to go to school for this' or 'you cant understand this at your age'
Because I can?. JK. Because that was my experience, of someone who is 2.5 older than 17?
> For some (many?) people, a 'proper' understanding develops _after_ the exploration.
I am afraid you have a too confrontational attitude here, but I'll answer anyway: because I do not believe you can simply "explore" such complex topics like building an LLMs. You'd simply be unable to build LLM drom scratch, unless you'd call cargo-cult chaining magic numpy incantations you've taken from Karpathy's tutorials "exploring".
If I were in "exploratory" state of mins, I'd rather go from entirely different side - I'd try playing with LoRA-ing existing small LLMs, such as venerable 2 y.o. Mistral Nemo, to get "feeling" for what training is and how hyperameters influence the process.
That's what I would call exploring.
I too started exploring programming as a teen by cargo culting. Fooling around and getting results is what made it fun. Understanding came later.
Then it is not "building llm from scratch" in my book. Just mindlees following instructions. Could be educational yes, but only trivially useful, if you have no bloody idea what you are doing.
> Fooling around and getting results is what made it fun. Understanding came later.
It is not how LLMs are "built from scratch".
But you're learning.
Maybe just accept that not everyone starts from fundamental theory, and there are lots of people who start learning by fooling around.
It is sold by PG as something special though.
> Maybe just accept that not everyone starts from fundamental theory, and there are lots of people who start learning by fooling around.
Even then LLMs are strange thing to advice to play with, when there are so much more interesting and theoretically accesible for a 17 y.o. so I wonder why would you'd particularly single it out.
I attempted many projects at a young age that I was absolutely not equipped for. The result of the attempts more often than not left me equipped, every time it left me better off. This is terrible advice.
Transformers are difficult to understand even to people with strong ML background, let alone a teenager.
That 17yo would have already built many uncommon bases, and would build further.
That is, a 17yo with proper mentality.
Can you provide an example?