I've worked with large MySQL databases that used row-based replication and things like an UPDATE or DELETE that affected millions of rows had to be applied in batches there, because otherwise one SQL query might result in a million updated rows needing to be sent to all of the replicas at once.
replyI've built custom batch-processors because percona-toolkit's automatic stuff was far too aggressive :|
replyEvery DB needs it, eventually. Even NoSQL darlings like Cassandra - I've seen it go into a resource-constrained death-spiral on stuff that should be async / non-blocking and safe. If you need to stay up, it's always worth planning on, and making sure your logic works during long-running gradual migrations.
Yeah, I think anyone that's done significant database work has come to the understanding that large updates need to be done in batches, otherwise you nuke performance.
replyOnce you get to about 1M rows of data, batching is essential.
> they just tend to make it less obvious you're doing something unperformant
replyIs this being positioned as a strength, in your comment?
It just is what it is. Sometimes you want to write the obvious query without the DB getting in your way, and other times you want to know as soon as possible that you're doing something that won't scale under exponential load. At this point in my career I prefer the latter, but the former will always have a special place in my heart.
reply