For file servers, I could be willing to give up 350 MB of memory temporarily for large-scale indexing and thumbnailing operations if it doesn't happen often and makes the file serving/browsing more responsive.
I've found jemalloc works best for long running libvips processes, fwiw.