Shouldn't it be within my right to withhold publication?
Or because I refuse to publish, but the publishers have a copy - I should lose the right to control my own work and the copyright falls and the publisher can just publish freely?
So in your hypothetical: you write the book and have a change of heart, I'll just copy it verbatim, slap my name on it and now I'm the author.
This would be wildly unreasonable and totally unethical. Every picture taken by a photographer has to be available on the market immediately?
Think through the ethical implications of that.