I think you meant to link this page: https://research.google/blog/a-decoder-only-foundation-model...
replyWish they gave some numbers for total GPU hours to train this model, seems comparatively tiny when compared to LLMs so interested to know how close this is to something trainable by your average hobbyist/university/small lab
replyEdit, it looks like the paper does
replyTPUv5e with 16 tensor cores for 2 days for the 200M param model.
Claude reckons this is 60 hours on a 8xA100 rig, so very accessibile compared to LLMs for smaller labs
That takes me to the same content as the submission, a GitHub repo (Chrome on iOS)
replyProbably the better link: https://research.google/blog/a-decoder-only-foundation-model...
replyAnd https://arxiv.org/pdf/2310.10688 if you want the full paper.
replyI suppose they tried to link this: https://research.google/blog/a-decoder-only-foundation-model...
reply