Are you sure that's true? If so, in which century did it start being true?
The difference between ownership of a physical object and ownership of an intellectual one is a matter of conventional. It's easier to define ownership of an object that is excludable, but that's human convenience, not a physical law.
Is that why it did fall apart?
Fighting physically for ownership predates fighting judicially for ownership.
To the extent that you can "own" another animal: the ownership of a female by a male is definitely a thing in the animal kingdom.
And before the first law was ever written, human slavery (estimated to be at least 4000 BC, with mentions in the first law ever written) did exist too.
Ownership predates the law that later on codified the concept of ownership.
Article I, Section 8, Clause 8 of the US Constitution
Which empowered Congress to "promote the progress of science and useful arts, by securing for limited times to authors and inventors the exclusive right to their respective writings and discoveries."
Scientists and the artists and their "exclusive rights" have built quite a lot over the centuries.
No, every time we do that, we do not give to the artist. But not giving is not the same as taking.
The obvious answer is that you take away a purchase the person to give the gift would have made. One could argue that there is also value in propagating someone’s art and potentially increasing the artists customer/patron base. Think of it as advertising or to put it in the context of a drug deal, the first hit’s free. The gift recipient may then go on to buy another work from that artist and even pass on the one they were given to someone else, continuing the cycle.
I’d also argue that there isn’t widespread agreement on reasonable compensation for artists. Personally, I don’t consider artists to be special enough in the context of people that make and produce goods, that they should get unique treatment. Why does a family deserve the financial benefits of trademarks and copyrights decades after the artists death. That’s just one example, but in a time when many’s artists view their livelihoods to be at risk because of AI, it’s not popular to engage in any debate that undermines the artist in any way.