(magazine.sebastianraschka.com)
Whenever I've tried to learn about LLMs, machine learning, etc. I always come out with a bad feeling, compared to learning about software.
Theoretical courses make me feel like it's all baby steps, unless you assume a level of commitment equivalent to a math degree. Practical/applied stuff seems to require levels of compute, training data, etc. that make it unapproachable for a home hacker for everything beyond basic exercises like a handwritten digit classifier.
Now this feeling of "impracticality" is perfectly fine. It would be the expected outcome if I was taking a course on classic literature or modern art. But machine learning stuff feels like it could have room for learning-by-doing applications that aren't entirely pointless, so I wonder if there's some way to approach the topic I'm simply not finding.
Did anyone have the same experience? any tips if so?
How many books did you read in your education? Far fewer that what LLMs read, no doubt. Of course we "have a life" and so we interact with hundreds of people, which is also a form of learning.
But in any case there's are fundamental differences here between these two systems (LLMs and humans) that we have not yet discerned clearly.
Say I ask it who are the most beloved Indian cricketers and why, it might not be correct but it will have the right vocabulary - cricket terms and gameplay I’ve not heard of, player names and styles I don’t recognise, things critics and commentators like and dislike about game strategies. And it is like that for almost any topic.
If they hallucinate a Python function, it will look like a plausible function e.g. “hn.create_user_account()”. They don’t hallucinate like
# create HN user account
# requires eye of newt and frog leg
“Oh great spirit of the internet I request and command you make a new account…”
We don't even need machines that can answer any question in a specific topic area. But we could use machines that can answer "I don't know." when they cannot compute the answer.
Consider the possibility of much-smaller LLMs that answer questions only in limited domains (physics, chemistry, math, medicine, etc.)?
Could not such LLMs exist? If not, why not? Would they not likely be more useful and far, far easier and cheaper to create and run (possibly even with far less hardware) than the current "know-all" LLMs? Also might they possibly avoid certain problems ( e.g., hallucinations, deceit, apparently undesirable behavior, etc.) associated with the "know-all" LLMs we now have?
One can train a decent model at home, for a specific task but you are absolutely correct that in order to have a model that's an expert in any topic it requires training and data on all topics.
I sometimes jokingly explain to certain people not immersed in tech that sometimes saying the "magic words" (commands) from the "codex" (script) in the right sequence with a lot of hand waving allow "magic" (applications/web apps/etc.) to happen.
It generally brightens a patients day and makes things relatable to those non-technical, children, the elderly, or the like.
TLDR: thanks for the chuckle.
In the software engineering world, we're accustomed to the reality that to do great work, all you need is a sharp mind and a cheap laptop. Better tools help, but aren't really a necessity. You could probably become a top-notch Linux kernel engineer with nothing but dedication and self-study, a $300 laptop and a shabby apartment with internet access.
You just can't really learn ML that way, just like you can't do it with banking, professional trading or medicine.
Maybe you are right! I'd like to hear the justifications more though.
But I can't do the same for things that excite me in this DL domain, even running pretrained models requires hardware that most hobbyists can't afford.
Cloud solutions exist but unlike traditional dev where we could test almost everything about the application locally, only thing unsolved was deployment and distribution in some cases, which is a sensible spend on infrastructure.
Compute is only an issue if you want to train large foundational models from scratch. You can easily train non-neural models on CPU, and fine-tune smaller models (say <4B) on a commodity GPU.
For training data there's lots of datasets online (like Kaggle), but you can build your own and can label things yourself or get a language model to label them for you. Getting good data is the hardest part of ML so this is a good exercise.
Then do some hobby projects: classify whether your photos contain a photo of you or a particular friend/family member, predict the expense category from bank transaction exports, classify your emails into categories, extract book recommendations from Hacker News, ...
For a junior coder, there's usually a path to recommend, joining some open source project or doing some home scripting, etc. Something where you can be useful even if it's not your full time job.
I'm wondering whether there's some similar hobbyist path available here, or whether there's too much of a jump between the educational/pop sci stuff and the professional realm. If it's the second, that's also fine, I just want to know if I'm missing something.
You don't need to change many weights, relatively, to add capabilities or modify behavior. Look up LoRas for instance.
I'm of the opinion that post-training existing models is an area with a lot of benefit and fun in the near future (it's already being done here and there of course).
I doubt you _really_ have to know everything in this course to know how to do that, but I'd want this as background personally for it.
You also don’t have to go straight into LLMs. You can learn most of the training mechanics via classic deep learning. It isn’t much different with LLMs except the training data and losses are specific and the models are much larger.
He recommends reading offline and code-free first (although chapter-by-chapter).
https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a...
...I wanna mention this book as well, although it doesn't fit the criteria: https://www.thelittlelearner.com[
But there is so, so much interesting work and research to do at a much smaller scale. I work on drug discovery models, and none of my work requires data or infrastructure that are not generally accessible to the public (it's nice to have the computing resources of my institution but there are other ways to access a GPU).
So if you are really interested in this stuff, I would suggest looking at the endless supply of interesting problems in domains that are not dominated by huge frontier labs.
You can google stuff to learn without the need to build a search engine too.
I have however learned a lot about math! And other related stuff. I'd argue if you extend your low level skills in math and code when the need for an AI comes up, simple weather / sales forecast, who knows, you'll have an easier time building one.
Don't try to pretrain from scratch. Take an already-trained Qwen and give it new powers.
That's what Karpathy's videos are for. You end up with a GPT-2.0 equivalent if you follow them to the end.
Opportunity knocks, but you have to build the door first.
For the rest of us, the sense of discovery that we get from training and running toy models is a big part of the reward. This stuff is not an exact science, at least not yet, so time spent building intuition with small-scale models is an incredibly important part of the learning process. Not pointless at all.
Compared to software coding, you build websites, apps etc. The barrier to entry is much higher.
I am trying to pivot into machine learning myself, so far my understanding is try to find a software engineer role where you deal with data, model infra, data pipeline, etc. Then within the same company, try to do more modeling work and build intuition and resume credentials in this area. Someone correct me if I am wrong.
The "baby steps" are not there just for intelectual satisfaction, the baby steps give you the building blocks to understand things that become progressively more complex. The baby steps are tools, that once you get familiar with them, you'll find new and new uses everywhere.
A good practical illustration of this: It is pretty hard to figure out how differential equations can be useful if you don't understand what they are. And yet, I bet that on january of 2020, there was at least one math teacher deep down in the amazonian jungle playing with a compartimental model based on a system of ordinary differential equations, to figure out the diverse scenarios this new epidemic could play out if that scary new virus escaped China and those few cruise ships.
Another example: take the concept of backpropagation: the underlying idea behind it is the chain rule from calculus, used to work backward through a computation and calculate how each parameter affects the error. Now, it is way easier to understand it in a small CNN that you are training to recognize handwritten digits than trying to understand it in the context of a full transformer model. A digit classifier is something that is way easier and affordable for you to fiddle with stuff and develop and intuition on loss functions, gradients, backpropagation and overfitting.
But that's not the point; the point is understanding these things better by seeing how they are made (or grown, if you prefer). Building something at a level further down the stack than you will ever go forces you to confront details that shape the layers above that you do deal with. This is one reason why Crafting Interpreters is such a beloved book here on HN.
LLMs are now at the core of tools we all use everyday... don't you want to know how they are made? Maybe not everyone has this itch to understand the platforms and tools they use, but I sure do, and it's helped me over and over again. Sure, LLMs aren't like operating systems or compilers. They're a quasi-organic mass of billions of parameters learned during training and honestly nobody (not even the mech interp folks) really deeply understands why they work so well. But that's no reason to completely give up and there is still a lot you can learn.
Take RLVR ("Reinforcement Learning from Verifiable Rewards"). This is the training technique that made LLMs get so good at programming and math in 2025 and 2026. A lot of people who confidently talk about LLM capabilities either haven't heard of it (and thus end up parroting out-of-date factoids about LLM training derived from the pre-2025 era) or have only a very dim idea of what it is. In this course you actually do RL training of an LLM to get it solving math problems better. You're right down in the details. It's interesting for its own sake but it will also give you a grounding for discussing questions such as: "why did LLM writing get so weird and bad in mid-2026?" and "why is it so hard to train LLMs to be good at things that aren't like math and programming?"
Anyway, Sebastian's two books are great and highly recommended; I worked through the first one a year ago and it was one of the best investments of my time in all of 2025. If you prefer video for learning this is great as well (although I'd still say - try the books first).
Sebastian, if you are reading this, this is an idea for another book: how to orchestrate agents for adversarial code review. I've been looking for learning resources to make something similar to this [1] at a smaller scale. I am aware of codex subagents, though. However, given that Sebastian already has a book for LLMs and another book on planning, I think adversarial code review is an interesting next step.
[1] https://bun.com/blog/bun-in-rust
[2] https://learn.chatgpt.com/docs/agent-configuration/subagents...
But this book is almost perfect. Every new concept is well described and every is well founded. And every step moves you forward (if you know, you know)
I've been wondering about getting a PC to poke around LLM stuff (not a practical use, but more for learning purposes) but I don't have any intuition about how much is enough and whether it is worth it to me.
> "the code examples have been carefully designed to run efficiently on a regular laptop, without the need for any special hardware. But if you do have access to a GPU, certain sections provide helpful tips on scaling up the datasets and models to take advantage of that extra power... The inference and evaluation chapters work well on CPUs and GPUs, while the training-oriented chapters benefit from access to a GPU if you want to reproduce the larger experiments."
So, consumer GPU helps but not essential. No need for expensive cloud GPU time.
He says:
> A reasoning model is essentially an upgrade from a conventional LLM, that is, adding more post-training via reinforcement learning with verifiable rewards (and optionally inference scaling techniques) to make it more capable of solving more challenging problems.
But this isn't really true is it? When reasoning models came around with OpenAI's o1, the main feature it had was to be able to try several paths and select the best one. This is what a reasoning model is. This was before agents and reinforcement learning with verifiable rewards wasn't it? And then RLVR and agents were added on later. Or am I wrong?
If you're truly going to build your own model (and commit to the cost of that), then video tutorials will not cut it.
And if you don't really, then I wonder if this is really the best style of conveying the information you'd be interested in?
I mean I might be wrong and maybe this helps emerging players or something like that, but to me it is more likely that this is more of a monetisation strategy of a personal brand. Living off selling to people that think they _should_ but then in the end don't.
It's also not a bad foundation for people who want to understand how these things are trained and work.
some people enjoy learning how the tech works
Maybe because they don't want to be an "emerging player" or something, they only want to understand how a new technology that is changing the world works and prefer to learn by doing.
If Sebastian is doing it by its personal brand, he really deserves it. I would trust him because I know he is a reference on his area of knowledge and I know he is because he shares his knowledge instead of only making fancy talks.
It's a critique of the people and the lack of substance. But not of the author. The author is highly capable.
That makes no sense. It's like saying if you want an engineering degree, going to lectures will not cut it. Of course not, you have to do the homework and projects too.
Something is off here. It indeed makes no sense
The target audience of this is for people that like to _think_ they're doing stuff by paying for it, to then not do anything.
It's a purchase of an identity, basically.
Which.. I guess is fine as something to cover Sebastian's expenses to enable him to keep delivering this free stuff. If you want to take money from someone, taking from this group is likely the most ethical thing to do.
What's next? Are you going to give us your opinion of music genres you don't ever listen to? And tell us who should listen to them too, I suppose.
How do you think people learn to build/train models? It comes down from the sky?
My core point I think is "first principles or nothing", but, thinking about what you said, I could potentially be off there.
I might be lacking the connection to and understanding of academia and how research and work gets done there. Maybe it actually isn't possible to do this work that way.
I doubt it. Strongly. But this is a valid counterpoint. (Which I had to kinda come up with myself, but anyway)
He is covering expenses. Probably barely.
It seems that you want to oppose something really badly today, but you're struggling to find something that makes sense for more than 5 minutes.
There are genuinely people who enjoy learning how something works by doing smaller scale versions of it themselves. This site happens to have a much higher average percentage of them than most other places.