I think this is a case where the theory of evolution feels pretty handy-wavy and doesn't answer the mechanism of how this happens. The article even talks about "bioengineered" silk! There are certainly lots of cases of evolution, so it is clear that things do evolve, but the problems of a theory are revealed in the edge cases, and I think this spider suggests that the theory really needs a lot of work to be a robust theory. I don't think a simple gradient descent (evolving toward more fitness) has explanatory power in this case. A theory that hand-waved around edge cases like this would never fly in Physics; at best it would be considered a holding theory until a better one could be found, like dark matter.
(Plus, I think it's fun to think about bioengineering in a sci-fi, or even Babylon 5 way)
First, ecosystems don't work on this concept of "balance" or "problems to fix". There are ecological niches that are exploited by a species or not. We have both generalists and specialists, predators that will eat anything and predators that have specialized in a particular prey or hunting approach. The world is full of organisms with remarkably complex behaviors, the field of evolutionary biology would collapse if this sort of behavior disproved it.
Second, there's really nothing hand-wavy about how specialized behaviors evolve, and in fact there is a mountain of research in the fields of evolutionary biology and behavioral ecology that have significant explanatory power here. I would argue that you're the one being hand-wavy saying "I don't think a simple gradient descent...". Evolution is not a simple gradient descent, and no evolutionary biologist would ever argue that it is.
This is the first spider web known designed to catch only one species of prey. That alone would make the finding extraordinary. The trap can lure only green ants and serve the food exactly were the spider wants it; granting access to a common source of food that is everywhere, but also that is very dangerous to hunt (as much big as the spider, with powerful jaws, and much stronger).
The video shows one most interesting thing: Notice that the spider is carefully moving out of the way, just a second before the ant is launched. The spider knows in advance that its current location is about to be hit by a bungee jumping ant, and acts accordingly just in time to avoid the "bullet". We can easily imagine the spider thinking 5,4,3... This means that spider brains can predict the future outcome of a complex movement of objects in the physical system of its trap, and also calculate how much time the fibers will resist the jaw of the ant.
They specialize is hunting spiders, changing hunting tactics based on type and number of prey. Yes, they count. They strategize. They make multi-step plans that take them out of sight of prey. And some people keep them as pets.
I can personally vouch for them being great pets. They're active during the day, hunt prey, don't need much food or water, and tend to "hide" in silk cases they build along the top of their terrariums so you can always see them. They like to get water from inside flowers, and probably can differentiate between many colors, so adding bright flowers not only makes things prettier, it provides a watering hole and possible hunting advantage over color-blind insects trying to hide.
Of course, if you don't want to set up a terrarium and personally sentence crickets to death, just look at the screens in your windows. Odds are, a jumping spider is already living there and will stay as long as you let it. They're territorial.
It's not moving before the ant is launched. It's moving as soon as the tension in the web is gone, ie. there's movement in their web. Most spiders react to movement in their web.
This is a completely automatic, unintelligent behaviour; if you remove the paralysed caterpillar at any point in the process the wasp simply goes on with its business (it will close its nest without any caterpillar inside, where the larva will die out from lack of food).
In the late 19th century, Jean-Henri Fabre studied these wasps (and many other strange insects) and had a copious correspondence with Charles Darwin on this very matter. His books are absolutely fascinating (ditto the letters Darwin and him exchanged).
> it will close its nest without any caterpillar inside, where the larva will die out from lack of food
...wouldn't the larva be with the caterpillar? You said that's where it lays the eggs.
1° the wasp digs a hole in soft ground
2° it paralyses a caterpillar (some other species use different worms, or beetles, as their victims; but each species of wasp targets one precise species of caterpillar or bug and cannot use any other food source for its larvae than the one it's genetically programmed to catch).
3° it carries the caterpillar into the hole
4° it lays its egg on the caterpillar, far from the head
5° it closes the hole with dirt
6° the egg hatches in the dark, and the larva feasts on the paralysed caterpillar, eating it alive.
If you pull out the caterpillar while the wasp turns around to lay its egg, it doesn't interrupt itself. IIRC even if you take the caterpillar while it's flying to the hole, it still proceeds to the end, without the caterpillar (which makes no sense at all, showing it's a purely reflex activity, without any intelligence).
Se for instance https://www.gutenberg.org/ebooks/3462
(Jean-Henri Fabre was a marvellous writer, and the English translation is very good).