4 bit quants can be fine, try it and see.
Please expand on this if you can. Why can't you? Who mandated this?
[0] https://digitalpolicyalert.org/change/20659-physical-and-cyb...
In fairness this doesn't warrant the way that the policy targets Chinese models specifically, but Chinese providers do fall under the broader category of entities that cannot be prosecuted by the U.S. law if such behaviour is found.
Alibaba (Qwen) at least has plenty of people in US offices to go arrest.
Yeah, it's trivially easy to rip the alignment out of the model, but that's only because it's tacked on as an afterthought. If they implemented their censoring as part of training, instead of inferencing, it would be extremely difficult to eliminate.
Alignment isn't only used for censoring, and it doesn't have to be transparent. Considering how heavy of a hand most governments put into companies developing frontier models, it's reasonable to assume they will start restricting capabilities, e.g. there may be a domestic version of Llama and an ITAR-compliant Llama, and China will likely do the same with Qwen, Deepseek, and the like.
Then it continues to provide the chain of events, the estimated number of killed protesters and so on.
I'll give you something to try: ask a western model about the Palestinian genocide. Or ask about the legendary Norm MacDonald 9/11 joke, which has millions of views on YouTube and how it compares to his other humor.
I think the root of the difference between us is that you are scared of their fascists, and I am scared of ours. Both are reasonable things, but only one of us can actually do anything about it.
With local models you decide how the firewall works. If you don't want it to internet, don't let it.
With US models you get whatever they say you do, and pay whatever they demand.