upvote

points

by pama8 hours ago |
comments
upvoteby scottcha8 hours ago|
next
[-]
We do use spec decode on all these models (I don't know if the paper is using that but I'm assuming not given the models listed) but not all these models have integrated spec decode stats in the responses. Turns out GLM 5.3 does though so here is a pass on that model.

   Language      Accept   Tokens/step   0/3 steps   3/3 steps   Output tok/req
   English       76.53%      3.296         9.4%        62.8%            549
   Chinese       73.37%      3.201        11.3%        58.2%            754
   Russian       73.88%      3.217        11.1%        58.5%            739
   French        75.50%      3.265         9.8%        60.9%            631
   S. Pashto     70.44%      3.113        12.9%        54.5%           1332
   Tigrinya      79.72%      3.391         8.2%        67.8%           3643
   Shan          75.64%      3.269        11.2%        63.0%           3675
   Tibetan       74.67%      3.240        10.3%        60.2%           2857
   Overall       75.75%      3.273        10.3%        62.2%           1772
reply
upvoteby pama7 hours ago|
parent|
[-]
Thanks. So the acceptance rates are rather similar across languages.
reply
upvoteby scottcha7 hours ago|
parent|
[-]
Yes it looks that way. I pulled the energy data for GLM 5.3 as well just to check the core findings are roughly the same and they do seem to be so spec decode doesn't seem to have the same divergence as the prefill/decode energy (which is rougly a function of thinking/decode lenghts and less a property of individual tokens).

  Language      Strict  Lenient   Cap   Tok/req     J/req    J/tok   Req x EN  Tok x EN  J/tok x EN
   English        100%     100%     0      548.7    230.33   0.4198       1.00      1.00         1.00
   Chinese         96%      96%     1      754.0    694.47   0.9210       3.02      1.37         2.19
   Russian         96%      98%     2      738.8    592.15   0.8015       2.57      1.35         1.91
   French         100%     100%     0      631.0   1101.84   1.7462       4.78      1.15         4.16
   S. Pashto       88%      98%     6     1332.1    208.06   0.1562       0.90      2.43         0.37
   Tigrinya        56%      62%    19     3642.9   1445.84   0.3969       6.28      6.64         0.95
   Shan            44%      54%    27     3674.8   3791.59   1.0318      16.46      6.70         2.46
   Tibetan          2%      70%    15     2856.9   5306.68   1.8575      23.04      5.21         4.42
reply
upvoteby vova_hn26 hours ago|
prev|
[-]
I think that the fact that TFA doesn't mention speculative decoding at all is not a good sign. It casts doubt both on the accuracy of the data and on the competence of whoever collected it.

I can imagine that in some cases the acceptance rate for non-English responses may be much higher, because while words on average need more tokens to spell, it is more "obvious" what the next token should be, so it is easier for the speculation model to predict it accurately. Information density per token is lower.

Or, maybe the acceptance rate is much lower, if the speculation model is tuned for English and doesn't work well with other languages.

reply