Okay, I tried it! It doesn't have a zero-shot mode, so the only way I could test it without training was with the SNLI head as judge. On all 5,240 prompts that stand-in ignored the number of people (pulling in about half the wordings whether 0 or 20 people were at stake) and contradicted itself 96% of the time. You could train a "trolley head" but that would just measure whoever labelled the training data!
replyAny relation to https://github.com/firelex/jeff ?
replyHasn’t seen it, authors not related
replyLooks very cool though. I’d definitely like to somehow “generalize” Jeffy to be more of a universal classifier. Although adhoc models pretty much always beat general ones at classification
running it now!
replyAwesome! Super curious about how it fails, and then figuring out how to over fit, err I mean improve it, haha
reply