upvote
> RTX 3090 for each and every employee

That’s quite inefficient. Get 2x RTX 6000 Pro and run DeepSeek v4 Flash for the whole company. This may or may not be cost effective compared to cheap third party providers, but it lets you own your own data, is not actually that expensive, and reports suggest that you can get 2k tok/s or so from a setup like this (at respectable batch size, but that’s exactly what you want in this situation).

I’m not sure I believe that the unquantized performance is as good as people are claiming, but even light quantization would make it fit easily.

reply
Depending on your compliance needs this may be the only way.
reply