How are your devices connected exactly? Using ethernet on the subnet your HA instance is on? And are you then able to also connect to a separate Wi-Fi SSID you create for those devices?
Also, are you able to join the Discord channel? Then we can create a thread and go a bit more into depth about your setup.
Right now, TOMMY only use Wi-Fi and doesn't support using ethernet for the communication with HA and Wi-Fi for peer-to-peer communication. But that is very possible to implement. Just haven't encountered the use case before now.
I'll probably implement an option for split communication (ethernet, wifi) and then allow users to decide the Wi-Fi channel devices communicate on peer-to-peer. That way you'd have full control over possible interference.