With BUILD happening tomorrow, I suspect Microsoft is going to have some stuff about local AI there with MS Foundry on Windows/Foundry Local. The timing of this announcement a day before BUILD is obviously intentional.
replySuddenly all the Windows K2 stuff makes sense, but I doubt it'll be enough. Its too little too late for Microsoft.
They've spent more than a decade cratering their OS. I doubt they will be able to turn it around with one code-named project.
replyI do. I can take my laptop anywhere I want, for example to a coffee shop and run a coding model while eating a croissant without worrying about an internet connection, as the term local model implies.
replyAnd you can warm up the croissant by just placing it on the trackpad while you wait for the LLM to finish
replyThe coffee shop doesn't have wifi?
replyIt's not very good and SSH is blocked.
replyI always use a VPN (to my own home server) for this reason (and other reasons) when connecting to public WiFi.
replyHow much does a dedicated server with 128GB vram cost a month.
replyHow well will the local LLM run when your laptop is in your bag while you're walking around?
replyYou can get an H200 (141GB) here for $2,700/mo: https://deploybase.ai/articles/h200-price
replyI could be wrong but my understanding is that 24/7 dedicated servers are wildly economically unviable. The reason cloud tends to cost less than local today (other than the subsidization) is because you aren't running models 24/7. So like 6 hours of cloud per weekday might beat the yearly cost of building local machines, but it's not in the same universe if you're running 24/7, as evidenced by two months of H200 rental costing more than the DGX Spark this Laptop is built out of.
I mean not that much? You buy the hardware once and then it’s just running for many years
replyLess than this laptop.
replyNo one is doing serious work on Windows anymore, those who are have yet to realise the clownshow they are collectively a part of.
reply