Which is pointing the finger at Sglang, who also weren't responsible for this.
replyAt a minimum someone should have realized during testing that their utilization figures are off by 40%.
A much better title would be "Cloudflare configuration error leaked data between user sessions"
I mean they are responsible for this. It's definitely a bug on their end.
replydeleted
replyDoesn't matter, they could have marketed it as 1 million token window, you would then infer, get to 600k and get an error. You won't be able to finish your work, and you won't get credit for 600k tokens.
replyThey thought it worked up to 1m token window
replywhy would they think that? I'm not a pro, but I know you can't serve more than kv cache pool that you have. If you want to run concurrent requests, you need to have more kv cache pool than your context window. The entire thing is weird, my bet is they were thinking most people will be using about 100k cache, compacting, etc and not nearing 600k ever.
replyBro, I had to read this sentence in the context of the previous one, 3 fcking times.. and it sill does not make any sense to me.
replySounds like they are saying they tested with a long prompt and just tested for a failure, but the test likely would have seen that same KV cache leakage and just interpreted it as success.
replythank you...
reply