The landmark study in humans for this is the Finkelstein 2013 paper [0] -- they gave humans Testosterone with and without AI to block aromatization to E2. In the AI group, sexual desire and erectile function declined markedly across the board, even when they were given high doses of testosterone.
Then you have studies like [1] and [2]:
> "Both estradiol (E) and dihydrotestosterone (DHT) contribute to the activation of mating, although E is more important for copulation and DHT, for genital reflexes."
> "We show here that a single injection of estradiol (500 μg/kg) rapidly and transiently activates copulatory behavior in castrated male quail pre-treated with a dose of testosterone behaviorally ineffective by itself."
The underlying theme is that across animal species, estrogens are regulators of sexual desire/libido while androgens support the necessary biological functions (erection) required.[0] https://www.nejm.org/doi/full/10.1056/NEJMoa1206168
[1] https://pmc.ncbi.nlm.nih.gov/articles/PMC1952538/
[2] https://www.sciencedirect.com/science/article/abs/pii/S01664...
There are two primary drivers behind why anti-androgens would cause loss of libido beyond effect on T:
1) AA's cause androgen receptor blockade systemically. This blocks action at the AR that would be residual across systems from adrenal production. Most important for libido are the AR activity that occurs inside of neurons and astrocytes in the brain
2) AA's have a two-punch effects on the Prolactin/Dopamine system + Progesterone system. Chronically elevated prolactin causes down-regulation of dopamine, which by itself is enough to kill libido. Progestins modulate GABA, which can cause "flat affect" and "emotional flatlining".
The combo punch of neuronal/adrenal AR blockade + Prolactin/Dopamine dysregulation + GABA dysregulation would require a miracle to have preserved libido on.