Commercial carriers such as airlines are also not allowed to transport people who need ESTAs/visas and do not have them to the US, so in practice it's relatively difficult to even get to the port of entry without a visa.
The visa is immaterial to the fine unless it conveys something beyond merely allowing you to present at a port of entry, which is allowed to basically everyone.