Publishing screenshots and the README on GitHub but not the code is sneaky at best, deceiving at worst.
I took it to 39% complete, as a language, over 8-weeks. Then, I spent just 50 minutes describing all the missing parts and it finished the language in 1-shot. I didn’t think it would work. It did.
I used Opus 5.5 on a $20 plan and I didn’t even hit my limits.
But we've also spend a lot of effort over the past year and a half to make the codebase and tooling more approachable to LLMs so they know how we like to do things and follow our standards.
To get a toy version that can run most things takes under 2 hrs. Half of that time is collecting the tests you want to pass, how it should evaluate the code, what your criteria are, etc.
Reinforcement learning doesn’t work by training an LLM on a series of inputs like ‘rewrite python in befunge’ paired with a prepared model implementation. It doesn’t matter if an open source implementation was in the dataset that trained the language model. All it got from that was an appreciation for syntax and language idiom; when it is generating code it’s doing so according to the approaches and patterns that it’s been reinforced into following in the context of a coding prompt - regurgitating existing codebases is undesirable so it is a behavior it will have been reinforced against.
Sounds like it's you who is behind the times. With things like REA we know that LLMs can reverse engineer the behaviour of binaries and basically treat them as a form of specification to rebuild an application.
This means that old python interpreter acts like a specification for what kind of features must be included inside the project.
LLMs are amazing at input to output transformations and that hasn't changed at all.
I think it's you who has to update his mental model away from "stochastic parrot era" accusations.
PS: if you're trying to accuse me of being a "stochastic parroter", that's exactly what I mean.
Taking an input and transforming it to an output is the exact opposite of repeating something.
https://app.preview.craftspec.com/app/city/
This whole thing is just a proof of concept, and needs a mouse to interact with.
It’s incredible what AI has made possible with experimental projects that would require months of coding.
There's so many youtube channels getting hundreds of thousands of views on EVERY new major model release just doing essentially zero effort one-shots, and people lap it up as if building things that already exist is somehow the best their imagination allows for.
I do love being able to test a model for 3+ weeks (>1 month is too long personally for my budget) and actually learning how to build scalable software and actually develop your "engineering" skills.
https://blueblazin.github.io/blockviz/
I tried it on some of my code and it's pretty neat, though not fully intuitive to follow more complex code structures for me.
Same, and I have found some of the failure modes in the realm of eyes free coding and limited auditing. I often see it more as pushing the process than the model, as I tend to bounce around models for the benefits diversity provides.
Have you also found there is a point where they hit code complexity and start churning without progress? (at which point you have to step in to help the bot step back and think big picture, which usually takes a few sessions)
I was motivated to make the codebase visualizer linked above to help me understand how the various pieces of the system that is being vibe coded connect with each other.
Now I'm thinking about how I can build something similar for agents. How can we teach "systems thinking" to a model?
I've been mostly working on very large modeling projects (think detailed architeture over 1000 acres) or animation because this really was the "new" capability area that Opus 5.5 "unlocked".
I have been surprised how well it has worked at scale with many sub-agents. However the code base is limited to the rendering engine mostly.
Certainly in the older models they hit a complexity limit, but I haven't really hit it with Opus 5.5. I have had it make data flow designs that I didn't like because they were more complicated than it should have been or the perenial problem of trying to make everything backwards compatible when working on greenfield refactors, but I haven't seen it choke on a code base yet.
To my experience, it's a simple complexity curve where the first bits go fast and the speed goes down very fast once you reach a certain level of complexity. Size doesn't matter as much as complexity: with the right foundations, it keeps zooming through it. Now I do wonder how much my inexperience in the complicated project is a brake on progress vs. it just being a hard to tackle problem.
On the other hand, I have a TS only project core is sub-20k LoC (NES harness), where it is struggling (for now, see other comments)
It helps us keep track of things, why not AI, too?
I've been working on process more than model. Similar to having a PR review team, you need multiple teams or processes that run over the code, markdown, assumptions. I still don't think this is enough.
I'm working on an NES harness, and the agent was churning with detecting certain state changes like dialogs, menu navigation, screen change. The bot and I got churning and I put it down for a while. A few days later it occured to me to ask the bot "Does the emulator and game state change with frames? Can we just get the state each frame rather than trying to be clever about it?"
It said yes, but added some nonsense about being performance hindered... (flash model). I haven't worked on it again yet, but my intention was to next have it build a small recorder based on this so it can get frame accurate game state change.
This is just a long way of saying, sometimes you have to step back and help them help themselves. They have a really hard time with recognizing ambiguity or how to determine lack of context that could be filled.