Yes it’s a real effect. I’ve just hardcoded offsets on various jobs:
reply * * * * * sleep 17 && …
That way I still maintain a per-job 60-second cadence consistently (which random would not), but don’t do it right on the top of the minute.
That's interesting as it is also visible with a crontab -l so that new scripts can avoid using the same second. However, if you have a farm of machines with the same hardcoded value, they will all still stampede at the same time just not at the top of the minute. By allowing a random value, their start times are slightly offset with in the minute. Is the precise 60s consistent interval really that critical or just that each machine checks once per minute-ish?
replyYes, using prime numbers are a good way to reduce the risk of multiple timers coinciding, for example a 10s, 30s, 60s period on a freshly restarted node / process.
replySystemd timers do this natively btw, no need to hardcode
And imho are also friendlier.
replyBut systemd bad :[
replyPeople slowly are getting used to it. (Less so with journalctl)
replyBSD cron has supported jitter for decades. This is strictly a linux problem.
replyIt's a human problem. Noone "expects" their own "minimal"
checks to compound. And everyone (re)implements periodic checks in ever-more-inventive ways.
BSD cron solves this ? Alright. Does prometheus use it to schedule its periodic polling ? Do OTel collectors use it to smear their backend metrics uploads ? Does your own code use cron jobs for its "periodic" heartbeating ?
replyIf the answer to that is all-yes, then my compliments to you. Alas ... at least my experience says that's not how things are done.
The Google internal equivalent to cron has a ? placeholder in the scheduling spec syntax letting the scheduler deterministically pick an arbitrary valid value within the field range. So ? 2 ... will pick any minute between 02:00 and 02:59 (and I suspect any second within that minute). The value is derived from a hash of the job definition, so overall this spreads the cron load within the hour.
replyObviously cron jobs are not all equal in terms of load and duration, and hours are not all equal in terms of usage, but that still helps prevent trivial spikes similar to the one mentioned in the article (disclaimer: I'm not aware of a recent counterfactual experiment proving this, but that seems a fair assumption).
That still lands it at the top of the minute, which is AFAIK the pattern the researchers found.
replyI suppose one would end up with checks happening on a random second instead of a random minute - reducing the size of the spike by a factor of 60, but still spiky.
replyYeah, and it's not cron, it's "everything running on a digital schedule"
replyAnalog systems naturally have a dispersion to them, digital ones, especially with everybody on NTP, much less so