Also, how would PRs and code review be handled?
Your suggestion really only makes sense for a small single developer hobby project in an interpreted language. Which, if that is what you intended, fair enough. But there really wasn't enough context to ascertain that.
If you're already at the point where you're fielding pull requests, lots of long running tests, etc., you'll probably already know you need more than git over ssh.
Yeah. You probably do want to make sure you turn your .git/ into a "bare" git repository but that's basically it.
And it's what I do too: an OCI container that gives me access to all my private Git repos (it sets up SSH with U2F so I get to use my Yubikey to push/pull from various machines to those Git repos).