It is a well-known trick, given that the timestep is between 0 to 1, you can slicing them at any resolution (1000, or 10000, give or take), and then keep a look-up table for modulation scale / bias etc for each. It is quite different from quantization and it is indeed lossless.
replyIt is also only applicable to diffusion models as only these operates at per-timestep.
So... why didn't the model ship this way to begin with? They just wanted to waste VRAM for fun?
replyThey ship a complete checkpoint for easily management (inference & training) in their own infrastructure. Moving to a LUT would make training on these layers impossible. BTW, these are not useful for lightweight fine-tuning, but might still be useful if you do serious post-training work.
replyOf course, these are also not an issue for things like FLUX.2 which adopts DiT-Air arch, that doesn't have this wasted space issue.
It may or may not be true. The people who made this modification and the other commenters didn't do anything rigorous to verify what they did. They just eyeball it. They could very well make some other error - this has happened frequently - that developing on prod, not knowing what they are doing, has and hopefully will again solve.
reply"Modulation weights" here refers to weights used to adjust layer normalization depending on the task (adaLN). General-purpose LLMs generally don't have those in the first place.
replyOne thing similar would be projecting both the head.weight and the final LLM activations into a smaller vector space, since that is basically just cosine similarity ranking step (so that would reduce the head.weight size). But again, it must be tried many times and just not working as well. LLM space is pretty saturated with tricks.
replyAlso begs the question whether this is applicable for high-throughput applications on FPGAs, which are to my novice mind basically LUTs, right?
replyI remember a paper which was posted on HN a few weeks ago where somebody implemented KAN networks in FPGAs, since those can readily be approximated as LUTs.
Whoah, could this mean we can treat layers like a jpg, where we come up with a formula that estimates the weight values of a layer instead of storing all of the weights?
replyThis is more or less what those lovely 1.5 bit quantisations are already doing
reply