I do something similar. I have all my esp/idf code in one crate and all my business logic in a separate crate. The business crate has no dependencies that won’t compile/run on the host. This is where all my tests live.

This is good advice however it should be noted that many of espressif's other purely software components depend on esp-hal, so you still want it to compile if you use any of espressif's provided components.