For someone who knows what they are doing, it's more like mosquito noise, a mere nuisance, but even then, using a rock solid system with all updates installed carries the risk of having a zero-day.
If your server is networked to the rest of the house, and if somebody manages to get in, then it's all fun(!).
Especially if you host something like wordpress with plugins you really have to be on the ball with updates.
That said, the practice of running a modern corporate web browser that auto-executes all programs sent to it from arbitrary unknown third parties is a way, way, way bigger and more common and likely attack surface than a simple static webserver serving files in directories.