I don't understand why he's citing a NeurIPS 2025 paper when cross-layer parameter sharing was introduced in ALBERT already in 2019.
replyif only tweeting is as rigorous as academic literature reviews
replySounds this like this will be a huge win for local models, since generally they're ram limited but have compute to spare
replyOnly if you double layers by layers instead of the whole stack (which IIRC is what nanbeige is doing).
replyTo put it simply, if you have 3 layers A-B-C then A-A-B-B-C-C requires more compute but not more memory bandwidth, but A-B-C-A-B-C requires both twice the compute and twice the memory bandwidth for the same token generation speed.