There's wanting to work a bit earlier or a bit later than everyone around you but there's a limit to how out-of-sync you can get.
The modern society started with the Industrial Revolution, when the people understood that there are some very tangible benefits when everyone starts working at the same hour, and started building around that. This building included discarding "local time" (an astronomical fact) in favour of standard time (social contract.)