Possibly in your specific application, usually there are a handful of options far less painful than a rewrite.

For the original issue of GC pauses, a narrow change is to move problem data to non-pointer-carrying types, or the bigger hammer of manually managed slices of those types. The second helps with fragmentation too. Some workloads can be split into multiple processes as a direct way to have smaller heaps. If none of those options are enough then off-heap storage lets you do whatever you want.

I do have some complaints about Go, but one of the big ones has been fixed since I last wrote much Go code and it seems like a fine choice for a lot of applications.