undefined

points

[-]

I was thinking this might be a result of the Cheap-money (post covid) era ending and everyone scrambling to reduce their Datadog/Cloud costs. Thinking back on 2023/2024, lots of companies were leaking large amounts of capital to those vendors and I imagine lots of people saw an opportunity for creating leaner and cheaper stacks.

by robertlagrant39 minutes ago|

parent|

[-]

This is my instinct too. I've had the pleasure of using DataDog and the pain of negotiating with their salespeople!

by yard20101 hours ago|

prev|

[-]

I have tried to self host grafana (loki prom and alloy) as o11y stack for prepbook.app. This is hard. I have a bsc in cs not that it says something. I managed to do it eventually, after some research. It was not plug and play in any way. The docs kept saying this solution is not production ready even. I couldn't find the production guide, only the "forget about self hosting and simply pay for us hosting this". After I deployed it the UX was so abrasive my partner won't even try to go into it to figure out a problem. It was a few months ago. Since then new solutions have arrived and I'm waiting to have the time to migrate. I saw PostHog have a solution but I prefer something I could self host and completely own.

I thought how come no one is trying to solve this problem. It looks like it's just a matter of time.

With that being said, my experience can be very skewed since prepbook is a passion project running on a VPS with essentially 0 scale. All I care about is the UX of the stack, not scale. Just for context.

by embedding-shape1 hours ago|

parent|

[-]

FWIW, I have no CS degree and barely attended school at all, and found Grafana + Prometheus + Loki fairly easy to setup, at least compared to what we used to use before those tools were available. Maybe it's because I used NixOS for the setup, but besides learning some new domain-specific things I didn't know since before, I don't recall hitting any particular bumps or roadblocks, I also went the 100% self-hosted route (spread across two hosts at home).

What exactly were you struggling with when it came to the setup? Just a ton of new concepts to learn which took time, or something specific to Grafana/Prometheus/Loki?

by dijit38 minutes ago|

parent|

[-]

"Getting it running" is the easy part.

"Getting it ready for production" is a different game.

I've fallen on my sword many times by trying to explain that prometheus fails every metric of production ready; in fact Google themselves replaced borgmon (prometheus) for Monarch because the "tiny unreliable time series databases everywhere" was in fact, not the successful and reliable deployment strategy that they had claimed.

But, it is very easy to set up. Just don't go looking for failure modes, because they're everywhere and every single one of them is catastrophic.

by denysvitali1 minutes ago|

parent|

[-]

There are ways to scale Prometheus (look at Thanos), but none of the solutions is really bug free.

See this PR for example (https://github.com/prometheus/prometheus/pull/18364) - this used to impact a production deployment I worked on. Prometheus, Thanos and even OpenTelemetry are full of those kind of problems - but at the same time it's the best we have and we should be grateful they're free and open source.

I'd still choose an open source stack (and contribute to it) rather than go for a proprietary solution - we've all seen what happens with DataDog & co

by ting049 minutes ago|

prev|

[-]

Do you think Prometheus + Grafana is the way to go?

by denysvitali5 minutes ago|

parent|

[-]

Really depends on the use case. Home lab? Probably.

Production? As soon as you scale you need a proper solution. Prometheus (by itself) doesn't scale - you need Mimir or Thanos (or similar)