upvote
Sounds like they've been set up as cron tasks. I've always been unhappy with the 1 minute granularity of cron tasks. Only thing I could come up with was putting a sleep(random()) type of line to avoid a thundering herd type of issue.
reply
Yeah, and it's not cron, it's "everything running on a digital schedule"

Analog systems naturally have a dispersion to them, digital ones, especially with everybody on NTP, much less so

reply
Yes it’s a real effect. I’ve just hardcoded offsets on various jobs:

  * * * * * sleep 17 && …
etc.

That way I still maintain a per-job 60-second cadence consistently (which random would not), but don’t do it right on the top of the minute.

reply
Yes, using prime numbers are a good way to reduce the risk of multiple timers coinciding, for example a 10s, 30s, 60s period on a freshly restarted node / process.
reply
That's interesting as it is also visible with a crontab -l so that new scripts can avoid using the same second. However, if you have a farm of machines with the same hardcoded value, they will all still stampede at the same time just not at the top of the minute. By allowing a random value, their start times are slightly offset with in the minute. Is the precise 60s consistent interval really that critical or just that each machine checks once per minute-ish?
reply
Systemd timers do this natively btw, no need to hardcode And imho are also friendlier.
reply
But systemd bad :[
reply
People slowly are getting used to it. (Less so with journalctl)
reply
BSD cron has supported jitter for decades. This is strictly a linux problem.
reply
The Google internal equivalent to cron has a ? placeholder in the scheduling spec syntax letting the scheduler deterministically pick an arbitrary valid value within the field range. So ? 2 ... will pick any minute between 02:00 and 02:59 (and I suspect any second within that minute). The value is derived from a hash of the job definition, so overall this spreads the cron load within the hour.

Obviously cron jobs are not all equal in terms of load and duration, and hours are not all equal in terms of usage, but that still helps prevent trivial spikes similar to the one mentioned in the article (disclaimer: I'm not aware of a recent counterfactual experiment proving this, but that seems a fair assumption).

reply
That still lands it at the top of the minute, which is AFAIK the pattern the researchers found.
reply
I suppose one would end up with checks happening on a random second instead of a random minute - reducing the size of the spike by a factor of 60, but still spiky.
reply
It seems more like one or multiple suppliers of battery systems not taking into account that the decentralized deployments of their system could have that effect. If it's not regulated and an open market, this was bound to happen sooner or later.
reply
In computer/systems/distributed engineering this is called request smoothing or traffic shaping, and it's implemented by adding jitter, which is a random time offset/pad added to the interval.

The funny thing is, modern electrical codes already account for this phenomenon. In order to implement Net Metering (selling your power to the grid), modern standards implement two kinds of jitter:

- Sandia Frequency Shift: a minute shift in the power frequency generated by inverters. If the grid is connected, this is effectively eliminated; if the inverter is disconnected from the grid, it detects the frequency has shifted and disconnects the inverter from the grid.

- Slip-Mode Frequency Shift: basically the same thing, but phase shift (which I don't understand).

Both of these are used for anti-islanding (don't create an island of power). When the inverter detects it's isolated and stops outputting current, it then has to check every 5 minutes, with jitter, for whether the grid is restored or not. It has to add random jitter, because otherwise all the mini battery/solar installs will come online at the same time and swamp the grid, causing another outage.

reply
power guys got surprised as rest of the grid load is human randomised and even huge factories will come on one chonk at a time, vs miilisecond co ordinated load switching at scale, with batteries as the load of the initial surge into even fully charged batteries can be almost infinite, and the whole calculation around grid scale turbines is that they run at very precise rpm's for stability and longevity, but worse, and a big fail, as battery storage is supposed to help with that, not be the problem. bet it gets fixed realy fast as those grid guys definitly get listened to.
reply
> grid scale turbines is that they run at very precise rpm's for stability and longevity

The turbines/generators set the grid frequency itself.

Having the batteries disconnect/connect would cause the generators to slow down and then speed up (or vice versa) which makes the rest of the grid frequency fluctuate as well.

It may also trigger the FCAS part of the grid that is paid to keep the frequency within the required range which is +/- 10mHz-200mHz.

https://mainsfrequency.com/frequ_info_en.php

reply
> as the load of the initial surge into even fully charged batteries can be almost infinite

There was an post a couple of days ago about a telco debrief of an interesting outage that happened. Several things went wrong like the audible bell was manually silence from being heard as a step in routine working but was forgotten to put back into normal mode once work was complete. The actual loss in power triggered the automatic switch to the batteries, but something else seemingly benign happened at the same time so that when power was restored, the batteries had been depleted to the point the charging process could not start because of something similar to this. I wish I could remember the write up, as it was really well written about the cascades of failures that occurred.

reply
I believe this is what you're referring to.

https://news.ycombinator.com/item?id=50153229&hl=en-US

reply
Yes, thanks for allowing me to offload my RAM to yours!
reply