You absolutely can just build things and put them on base images without building or pulling the base image at any point. This is a central feature of ko, a simple container build tool for most Go applications: https://ko.build
There are many ways to achieve this. If you are just copying artifacts, and not using RUN, then there is in principle no need to download the base image at all. Feel free to message me if you'd like to chat (email is in my profile).