You raise a good point. This is why people sign the checksums. The signature confirms that authenticity of the checksums. That somewhat moves the goalpost, though, since it then depends on where you got the source's public key, but it is still a more secure practice overall. The advantage of having the public key is that you only need to get it once and you can check many downloads later.
It is also possible to have a signed file that you can use to check the authenticity of a downloaded file directly without having to use checksums. Rust [1] does it that way for its other installation methods.
[1] https://forge.rust-lang.org/infra/other-installation-methods...