But in my opinion if we try and make life comfortable and easy for everybody, it will break the rest of society to try and achieve it.
I think many would argue that a society unwilling to feed or educate children is pretty "broken".
Is willing to make sure kids are fed bad, but making sure that kids aren't physically or sexually abused fine? Or is that also off the table because we don't want to make life too easy for everyone?
Because it seems that you're suggesting that thousands (and perhaps two orders of magnitude more than that) should just...grow up malnourished, underfed, lacking the warm comfort of security that is critical for healthy adulthood free from anxiety, depression, repressed rage, emotional dysregulation?
I'm sure you wouldn't feel the same way if you were that starving child or grew up with imperfect parents or in an environment that had actual hardships. This lack of basic empathy and introspection can only occur from someone who has been completely isolated from much of the real world their entire life. You are not arguing against perfect being the enemy of good, you are arguing against having any sort of good in the world at all, and it's obviously coming from a place of extreme comfort and luxury.
There's no chance you'd be saying anything like this if you had ever experienced anything close to poverty and starvation. Ironically, it appears having an upbringing that is too comfortable is what causes these beliefs in the first place.