I wonder how this would stack up against 4x RTX 3060, assuming you have the physical room for them.
replyI understand the B70 is a bargain vs AMD and especially nVidia offerings, but to me it feels like I would be buying something that would feel too limited in less than a year. 48G would be much more confortable.
replyAnd I know the 96G nVidia cards are selling for over 10k$.
The future can’t arrive fast enough!
You can buy two b60s for $1300 right now (650 each) if you want a total of 48gb. Intel recently raised the price on all of their gpu's except the b60 series, so they are currently the best deal per gb I think.
replyThey have terrible cross connectivity, it's not wise to share a model between them
reply32GB is perfect for models around 30B parameters. Since qwen has really hit the spot with their 27B dense models, I think it's a good bet. Also, 32GB is enough for other tasks such as image/video generation and loading multiple smaller specialized models
replyI have a B70, what llama options are you using and what performance are you seeing?
replyYou don't need $10k to run DeepSeek, I run it on a $1000 system.
replyGo on… what is this $1000 system?
replyCould you elaborate please? Genuinely interested
replyOh, can it work with the /v1/completions/ auto-complete endpoint?
replySorry, I wrote autocompletion by force of habit. I simply meant it can complete code you have already created a structure for, which personally is very nice
replyI thought so, but thanks for the clarification. I am a little bit disappointed that local autocompletion models have been left by the wayside in favor of models post-trained for agentic coding. Both Codestral and Qwen-2.5-coder are more than a year old at this point, but local auto-complete seems to me to be such a great usecase.
replyThe latest Qwen models (including 3.8 27B) do still support FIM-style in-editor code auto-completion if that's what you're looking for. I wouldn't want to use a large dense model like 27B for such a task (since FIM-style auto-completion really works best with low-latency responses), but it works.
reply