I believe the client is open source and there's a reverse engineered server (that some tail scale employees contribute to)

Headscale is an open source alternative, I haven't read the code but it might be a good place to start: https://github.com/juanfont/headscale