Having no swap limits how much you can overburden your computer, but you also hit problems earlier. Here's some example numbers for 64GB of memory: With swap you can go up to 62GB of active program data (85GB allocated and used) before you have performance issues. Without swap you can go up to 45GB of active program data (63GB allocated and used) before you hit a brick wall of either thrashing or killing processes. The no-swap version is better at maintaining snappiness within its happy range, but it's a tradeoff.
Also, that separate policy shouldn't even exist. For LRU/active-list/inactive-list purposes, why does it matter whether a page is anonymous or file-backed? If you need it, you need it, and if you don't, you don't. No reason for anonymous and file-backed memory to be separate sub-sub-systems under vm.