You are aware that packets don't magically appear at the server side when sent by a client, right? All packets have to be routed to the destination by several routers. All these have to understand the full address to route the packet. The IPv4 header is strictly defined though. It says 32 bits for the source and 32 bits for the target. If you change anything about that all IP parsers will go haywire. If you put the information somewhere else, every router that doesn't understand that will send it somewhere else.

Every client, server, and router, every device that uses the address needs to understand where it comes from and where it's going. That means all the software needs to understand the protocol. So instead of having incompatible implementations live within the same protocol and make a lot of chaos it's better to have a new separate protocol that can be implemented gradually. Now the distinction is between having or not having IPv6 connectivity and my package on IPv4 goes no where because it hit a router that doesn't understand the extension.