Are you sleeping enough? When I was getting too little sleep, averaging 5.5 hours per night, this described me well. A single sleep interruption could make me lose most of a day of work. I'm sleeping better and longer now, and it seems I'm more able to tolerate small interruptions.
Similar to you, I also noticed that if I miss good sleep for several days, it stacks. I treat sleep like a battery. A day uses up 20% and good sleep fills it back up, but only like 30%. One missed night isn’t that bad but I also can’t recover several nights’ worth.