I see a lot of back and forth about postgres' suitability as a queuing system. I wonder if there's a couple of separable problems here. Postgres backed queues - even very scalable ones - might work well for background jobs in a monolithic app backed by a single DB. Things like backgrounding sending an email etc.
replyBut usually when I reach for a queuing system, it's because I want to decouple a part of the architecture. And in that case, it's probably better to use a dedicated queueing system instead of postgres.
I wonder if the two cases are conflated in a lot of online discussion.
We are very heavily using Postgres as a queuing system in production for many years already, not just some quiet background tasks but with millions of tasks in the queue at any given moment. I have yet so see any issues with that so I'm always a bit suspicious when people say they had to reach for something else unless you are at a crazy scale.
replyI know it's very hard to compare workloads, but famous recent example: https://openai.com/index/scaling-postgresql/
> It may sound surprising that a single-primary architecture can meet the demands of OpenAI’s scale; however, making this work in practice isn’t simple.
I have the impression that SolidQueue uses whatever your rails app is using by default. That can be SQLite, PostgreSQL, MySQL, etc.
replySo I don't think they changed to PostgreSQL per se.
Correct, what I meant was that SolidQueue was the first database-backed default adapter over the usually common Redis + Sidekiq combo for ActiveJob.
reply