How is it crazy? It's perhaps not granular (the repository is the boundary, and that's that), but you can definitely restrict who can pull or push as easy as you can make rules for SSH.

Plenty of not-very-granular "enterprise" systems out there, it's not exactly unique to not always have full ACLs on the smallest of objects.

I guess it's because git wasn't developed as enterprise software.

No.

The maximum granularity is also 'per repo'.

git repos viewed through gitlab's slow Ruby monolith are still git repos, so it doesn't make any difference.