Society broadly agrees, enough that it's illegal in the US to stop someone just for "unusual behavior." You have to have an actual concrete reason to suspect someone of a crime. Not that police always follow the law on this.