Also, how would PRs and code review be handled?
Your suggestion really only makes sense for a small single developer hobby project in an interpreted language. Which, if that is what you intended, fair enough. But there really wasn't enough context to ascertain that.
If you're already at the point where you're fielding pull requests, lots of long running tests, etc., you'll probably already know you need more than git over ssh.