But that only specifies the decoder.

The format for all modern video codecs is not the kind of format where any specific piece of uncompressed input should always be encoded the same way, but more like a very restricted programming language that gives the encoder a lot of tools to compress the video, and which tools they use and how they use them are up to them.