You can still benchmark say 100 requests and time that. For JVM for example you'll have to consider the JIT effects and GC and stuff that only happens later. So 100x 1ms requests is still a very small benchmark that's probably unreliable