Yeah, I'm pretty sure I would get call for a quick chat with HR if I walked around the office with Mein Kampf under my arm as well.
Schools, prisons, etc are institutions that can decide what is appropriate for their audience and how they want to spend their shelf space. A real banned book is a book that gets you in legal trouble if you're caught with it.
They are banned in the US in the same way playboy magazine is, they are not allowed in certain places.
Would you say that Playboy Magazine is a banned book becuase its not allowed in schools and prisons?
I would guess the person you are responding to is thinking about more mundane books that are or have been banned in public and private schools in recent history. Harry Potter, for example.