You can't claim it "works" if it hasn't produced any coherent responses and is still early in your first training attempt.
replyIt is a goalpost that is easy to move. By "works" I mean learning from a continuous single (meaning batch-1) stream of data. The fact that it produces full words and full coherent phrases instead of a random stream of characters that would any typical LM produce if trained under the same training regime.
replyI would be okay if you shared it as a potential idea and possibly interesting early result, but the language you are actually using to characterize it is misleading or delusional.
replyPlease get a model to the point where it seems like it has some natural language understanding and then share again with reasonable characterization.
Is "AGI" the language that bothers you? Well, one has to keep his eyes on the prize and I see a bright idea which could lead to AGI, so, why not describe it as such? I also see the inspiration and hard work necessary to move that idea further along, so fingers crossed.
replyFor sure. As it will pass through the whole corpus I will share the weights, run it through established benchmarks for small models and share all of this as an update. I am also planning on making a Youtube video explaining in detail how it works on a deeper level and the whole reasoning behind why it is built the way it is. But no promises here.
replyNothing to be ashamed of if you end up pushing back the release date of your feature film :)
replyI had ideas not completely unlike this so long ago, but one big difference can be summed up in one of your parameters.
>Directories are walked, binaries are skipped . . . and each file is read from its beginning to its end because a document has an order.
For me it was binaries being walked because text and anything approaching a language model was so much further out-of-reach having such limited computer power.
deleted
reply