I think you should try to go back to school for physics. Or teach yourself physics. If this idea still draws you, you should do it.

What sub-field were you going for?