How sure are we about this? How certain are we that those specific species of mold have a net negative effect, rather than a net positive (like for example mushrooms)? Penicillium grows on stale foods and I doubt eating it would have a net negative effect.
Feel free to eat it.
"Penicillium Species and Their Associated Mycotoxins" - https://pubmed.ncbi.nlm.nih.gov/27924532/