undefined

points

[-]

How do you figure that?

When they just loaded the weights alone, it was taking 156GB in vLLM. After warm-up and adding a KV cache pool, it took over 200GB.

And this implementation is already cutting down the 1M token context window you would normally get.

by Tepix2 hours ago|

parent|

[-]

For sure if you want to properly utilize the model with several users in parallel and large context you'll want two MI350P.

by craftkiller4 hours ago|

prev|

[-]

Just want to add that while the MI350P is a PCIe card, it is designed for servers. It just has a heatsink (with no fan) which the powerful full-case fans of a rackmount server are supposed to cool. So while the MI350P is certainly more attainable for us regular folk due to its formfactor, we won't be able to just drop it into our gaming PCs like a regular graphics card.

That being said, if you're dropping tens of thousands of dollars on graphics cards then picking up a rackmount case to go around the card is pretty insignificant.

by cyberax3 hours ago|

parent|

[-]

Just put a fan on it. It's just 600W, so nothing super-special is needed. Or add a water cooler.

by monster_truck3 hours ago|

parent|

[-]

You're going to need at least a couple loud ass IPPC 3000s to usefully move that kind of heat if you don't want it to throttle. And then another normal sized fan for the doorway of the room it's in.

Not exactly super special, but a ~constant 600W+ of heat tends to be a learning experience. It's worse than a high end gaming rig, much closer to a literal space heater. I don't work during the summer because it sucks fighting both this and the sun with AC. I do have a fan that slots into the window and can push or pull, but kicking the waste heat outside doesn't help when its humid.