No, you can't divide the entire size by the expert count. A lot of weights are constant for all tokens, so total active count is ((2800-(shared)/896)*16 + (shared))
replyTIL, that makes a lot of sense, and thanks for the correction.
replyJust to add to that, a Transformer block consists of an attention part followed by a feed forward part. MoE only modifies the feed forward part (which basically contains declarative knowledge getting injected into the residual stream).
reply2.5x the scaling efficiency, so 4 times the price? What is happening here? Did the subsidies dry up with the discrepancy between chinese and US models?
replyScaling efficiency simply means if you took the first small model and scaled it up to the big model it would take 2.5x the resources to run. Not the that larger model is going to be any cheaper.
replyKind of like scaling your personal automobile to the weight of a semi, the semi is still going to be far more efficient in moving cargo, not that the semi will cost the same to operate as the original car.
It's also 2.8x parameter count (1T -> 2.8T), likely higher activation per token (50B?).
reply