When I'm listening to music, the music helps form my sense of time. It is deeply jarring to have the music pause for a few minutes and then start as though in 'music-land' no time had passed.
I'd be happier if the music volume went to zero but the song/track kept progressing.