but this breaks the entire premise of the agent. If my emails are fed in as data, can the agent act on them or not? If someone sends an email that requests a calendar invite, the agent should be able to follow that instruction, even if it's in the data field.

That sounds like an excellent idea. That still leaves some other classes open but it is at least some level of barrier.