You shouldn't need to B64 encode for transfer - piping over SSH is binary safe. Verifying with 100Mbyte of random data:

    user@host:/tmp$ cat rnd.file | sha256sum
    71d3392be59210c517ab889fca715a46dee54f6326a9d901db0addd351607cb6  -
    user@host:/tmp$ cat rnd.file | ssh user@otherhost sha256sum
    71d3392be59210c517ab889fca715a46dee54f6326a9d901db0addd351607cb6  -
    user@host:/tmp$ cat rnd.file | zstd -z | ssh user@otherhost "zstd -d | sha256sum"
    71d3392be59210c517ab889fca715a46dee54f6326a9d901db0addd351607cb6  -

When using zstd (or anything else) make sure you don't have something in your .sss/config that would make ssh use a compression as you'll waste a bit of CPU to send a little more data as a less efficient compression method is applied on top. Though TBH I think that unless I'm on a very slow link, if I'm sending something big enough to need compression it is probably in a pre-compressed or otherwise uncompressable format anyway (video, photos, …) so zstd does little, the one exception I can think of being if I'm throwing an unencrypted VM image from place to place.