undefined

points

[-]

A counter anecdata. We transitioned from a postgres job queue to Rabbit. We had never ending problems after that, many of them were misunderstandings, some where just wrong-fit. We migrated because we had some time on our hands and thought we would alleviate some high pressure jobs. Not only did it not solve the problem, but having written all the code that decides when to pull the next message and what to do with it, and how to dead-letter it - just worked great for us on Postgres. It was so easy to understand and doing things like reprocessing just using a standard postgres DB interface was much easier.

Ultimate the entire processing got removed from our team and no longer needs to do these deployments (acquisition transitions)...

by dorfsmay9 hours ago|

parent|

[-]

I tried very hard to use postgres as a queue, it was robust but slow once I started to push from more than a few processes/servers. Moving to zeromq initially and sqs after solved all my perf issues, and was still solid.

by vjvjvjvjghv4 hours ago|

parent|

[-]

I think that’s the way to go. Start with Postgres and only if there are problems, then think about something more specialized. Same for microservices. Start simple and introduce a service when really needed.

I hate it when people already start out with 10 or more different systems/services for a few messages per second.

by egwor58 minutes ago|

parent|

[-]

I think that using the right tool for the job is important and saves a lot of time in the long run. There are expensive headaches that we have to resolve.

by cheikhcheikh1 hours ago|

parent|

prev|

[-]

it's not the way to go if you hit the limits very quickly and have to waste immense time migrating.

by raverbashing1 hours ago|

parent|

prev|

[-]

Honestly Rabbit sucks more than it shines

Also it is very "unconventional". Everything has to be done in its weird and quirky way

by tensor15 hours ago|

prev|

[-]

Having done that, e.g. used rabbitmq plus postgres, honestly I wish I had just used postgresql for both messages and data. It would have been easier to manage by an order of magnitude, especially at scale and needing to satisfy enterprise requirements. Also the flexibility of postgres would have solved problems that we ran into because of limitations of rabbitmq.

by majormajor15 hours ago|

parent|

[-]

Without knowing any specifics of your uses, my usual starting point on that sort of design is that "messages AND data" is it's own special little way of ending up with a hard-to-debug-and-operate system. ;)

It's very hard to best-of-both worlds event-driven system + RDBMS-storage, it's very easy to end up with worst-of-both-worlds. Hello distributed transactions!

Again, you just should think about all the ways you want to use it and the maintenance/uptime requirements your users are going to have in advance.

by tensor14 hours ago|

parent|

[-]

I think messages + database are extremely common in any sort of large application where you have data processing nodes. Postgresql actually has very good mechanisms to support message style communication, and as long as you design your message tables independently you shouldn't have horrid issues around locking and transactions. Message queues don't save you from thinking about that anyways, they just replace transactions with acknowledgements.

Trying to manage a highly available and durable rabbitmq or other message system that can also be recovered from backup to an offsite mirror infrastructure in the worst case is actually incredibly difficult. Usually these systems are designed with the assumption that you can just regenerate messages based on database state anyways in worst case scenarios.

In this use case your database already is highly available and can recover on an offsite backup if you have suitable wall shipping going on. So you've done all the hard work once, may as well reuse it unless you truly have some mind bogglingly large message throughput needs.

Finally, we had a need of a queue that was more than just first in first out. We wanted to fairly balance workloads across users and tenants. Whenever you have such a need postgresql lets you design this type of queue far easier than trying to do some elaborate multi-queue setup with a traditional queue.

by cpursley15 hours ago|

parent|

prev|

[-]

And now there’s pgmq

by ivolimmen6 hours ago|

prev|

[-]

I once needed to maintain an application written in everything Oracle. If I ever encounter the original author of that product: I have things to say to him.

We quickly replaced part by part by easier, less costly parts.

Software development is not just writing code; I think all HN users know that.

by pbreit17 hours ago|

prev|

[-]

I dunno. I think the main takeaway here is that you can do 80-95% of your stuff in Postgres and eschew all the unnecessary, unproven stores.

by jghn9 hours ago|

prev|

[-]

It’s also entirely possible that nothing you do in the eventual history of your company hits a scale where this matters.

by AdieuToLogic8 hours ago|

parent|

[-]

Sometimes to scale is "continue to satisfy SLAs as service usage increases" and other times to scale is "successfully evolve functional capabilities over time."

While the GP may have been referencing the former, embracing "PostgreSQL for Everything" often prohibits the latter.

by groundzeros20157 hours ago|

prev|

[-]

> in painful centralized bottlenecks.

I find the opposite to be true. I cut out the decentralization and get it all one one machine, and the bugs go away and the perf improves.

by encoderer18 hours ago|

prev|

[-]

It doesn't take very long (because compute and storage are separate in most of them) but good lord does it get expensive. Every time you click that upgrade button you are doubling your cost. It's really painful when you have a spiky workload that is performing fine like 95% of the time but you are watching the p99 and need to double the cost of a very expensive infra component, only to improve the experience of the heaviest 4% of your workload. This is to say nothing of the gambit you then have to play with reservations/prepays.

by majormajor15 hours ago|

parent|

[-]

I haven't seen a way to get guarantees of upscaling operations under like 30 seconds (with Multi-AZ RDS) with well-supported RDS stuff (leaving out active-active setups with logical replication because that's a whole other can of worms).

If you know you're gonna be ok with that for a long time, go nuts. I'm just saying: think about it in advance!

The cost pain for spikes is also a thing - some of Aurora's billing models look potentially promising but I haven't used them in practice - though it's also somethings that's harder to avoid with alternatives. Distributed DBs aren't generally super friendly to dynamic scaling IME.

by cryptonector7 hours ago|

prev|

[-]

> If you start here, with the "Postgres will take you wherever you need to go" meme, without thinking extremely deeply about your schema and how you expect to evolve it in the future, you can easily paint yourself into a very difficult and expensive corner.

Yeah, backwards compatibility is not a thing for Java, Rust, C++, etc. :eye-roll:

Meanwhile in SQL if you need to make a backwards-incompatible change to your schema you can always use VIEWs and INSTEAD OF triggers to maintain backwards compatibility for code you've not fixed yet.