Thanks! I had a clanker graph it with sliders, its a very interesting function to play with! seems like it has the negative 'hump' typical of a GELU coupled with the saturation of a sigmoid.

Its interesting that saturation is ideal at this scale, i thought we were all in on self-activating functions (i.e. swish) but in fairness all those papers lacked ablations at scale nor discussion of stability.