The tax office can decide that the company is, in fact, a German company if the founder lives and operates in Germany. Now the company paperwork has to be done in two countries. I don't know what the tax implications are.
Applies to all EU countries btw, you can't just choose a different jurisdiction for your new company risk-free.