undefined

points

[-]

Google had done much to lay the framework for current models (and is currently doing good stuff with Gemma). But after ChatGPT took off I don’t think Google continued to release any big models freely before meta released its models (maybe it was because Google didn’t have any models at all during the time)

by kamranjon15 hours ago|

prev|

[-]

I honestly think of the T5 model family to sort of be the real beginning of this open model craze - I know BERT was already popular for classification etc, but T5 was the first sort of generally useful model, was exceptionally simple to fine-tune, and is still in use today (t5 base is still averaging over a million downloads a month on huggingface), has tons of variants and sort of kickstarted this whole community. US labs get a lot of flack but Google has been super supportive and open in a lot of ways that has pushed this whole endeavor forward, even if I feel like they've sort of declined in transparency in recent years with their open models.

by Topfi12 hours ago|

parent|

[-]

Couldn't agree more and can only recommend T5 as a base to anyone. It's amazing to get started, whether as a learning resource or for real (albeit very tailored) applications. Especially the BigScience fine tunes are such a great starting point and I, as a total layman, have learned a lot, especially concerning how a model can be optimised via all manner of methods since even mt0 is small enough to where one can do multiple runs with wildly different outcomes in quick succession. Quantise, prune vocab, try different approaches to sourcing training data, retrain dozens of times, it's all pleasantly possible on consumer hardware [0] and surprising how much you can squeeze in functionality-wise. How does latency change vs memory usage, what affects format reliability, how languages and scripts affect training and the efficiency equation, etc. are all quite exciting to learn.

Understand why T5Gemma is no longer under Apache-2.0 and honestly, have not seen that much advantage when testing that vs T0 in my experiments either way, but still, there are good reasons why plain old T5 and its descendants are still popular, licensing being among them.

Gemma team also has very consistently interesting models, especially like DiffusionGemma. Ironic, as (beside 2.5 Pro), I have never warmed up to the Gemini series of models but rate Gemma models far higher than e.g. Qwen in direct competition. In any case, thanks to the teams behind these for making as much possible.

[0] As in proper consumer hardware, not a cluster of DGX Sparks or Mac Studios solely for experiments that sometimes are asserted as being consumer grade...

by curious_cat_16311 hours ago|

prev|

[-]

Thanks!

When is Gemma5 coming out? :-)

by optimalsolver2 hours ago|

prev|

[-]

Hey, with the changes at Deepmind, is the Gemma project still ongoing?

by elliotec2 hours ago|

parent|

[-]