For context, the Python workload was a general-purpose fileserver with file indexing and image thumbnailing, largely IO-bound. Switching from mallocng to mimalloc resulted in a slight speedup (150% of baseline), but over twice the memory usage, going from 250 to 670 MiB, primarily for thumbnailing. Some pathological cases (libvips calling imagemagick to decode heif) was a 10x multiplier.
replyThere was multithreading, and yes, mallocng visibly became a bottleneck beyond 5 threads. However already at 3 threads there was diminishing returns for both allocators, so this was not an issue in this case.
The speed gain was hardly noticeable in practice since the program was already plenty fast, but the additional memory usage was a very real inconvenience.
Is that memory increase seen in peak usage or average/idle usage?
replyFor file servers, I could be willing to give up 350 MB of memory temporarily for large-scale indexing and thumbnailing operations if it doesn't happen often and makes the file serving/browsing more responsive.
On the other hand, plenty of NAS/file server systems are going to have paltry CPU and memory. Running slower with a more modest memory foot print might be the only way to keep the system stable.
replylibvips shouldn't be calling imagemagick for heif decode, it has a nice one built in. Unless you were using a very old libvips!
replyI've found jemalloc works best for long running libvips processes, fwiw.