The problem is in the rewards, but it's more how they average across domains and contexts, so AI misapplies them. Actually, in human preference evaluations, most of Gen Z in particular extremely adverse to the common look of AI images.
So it's not the annotators involved in RLHF, it's the models failing to map to human preferences at the task level. The failure rate is also uneven across domains.
Where there are challenges with human annotation, it's not the ability to gauge what's good but rather how to correct what's wrong. Most people can tell when something is off but struggle to articulate why. That's where more domain expertise is required.
For posterity, the memetic ugly food can be controlled and mostly filtered out with prompt engineering. Here's a quick ad hoc test example which (hopefully) uses the new model: https://chatgpt.com/share/6aebf030-9d38-83e8-a103-3509883905...
First image is fine, but attempts to make it more wacky turn it to the AI food slop. So then followup prompts deslopify it, mostly successfully.
I think the way they'll solve it in the end is to create an ultra-high-fidelity simulation of the world including modelling taste and other senses on the level of physics, and then incorporate all of this into the training rewards.
I think this is how the machines know what tasty wheat tasted like.
This kind of content is here to stay and we’re going to see more and more of it. Unfortunately…
I see restaurants post AI flyers and ads to random local Facebook groups and most of the replies are about them being AI slop. I think it’s an uncanny valley problem: People don’t mind if you post a real photo of your food or a cartoon drawing of a pizza or a pig on a spit, but when it’s a weird fake photo with texture issues, it looks gross, especially when it’s food.
I think a lot of people who open restaurants probably aren’t extremely online, partly because it would be physically impossible with the amount of work they have to do, but they definitely pay attention -sometimes too much attention — to public online feedback.
I absolutely do not get the uncanny-valley images. Those all look gross. Besides not actually being representative of the dish, the images have all sorts of weird texture issues, look too glossy, and are just unappetizing. I cannot understand why a restaurant owner would want to put up an ad that has food that looks straight up gross.
I've been seeing "AI slop" stickers posted on things
> I see just everywhere including food menus
The places I have seen it in use are the places that would have a handwritten scribble that was misspelt and would be something mocked by people because it was written by ESOL (or vice versa, English speakers trying, but failing, to write in a different language)
Honestly, I think the AI slop is a step up - I can read the signs clearly, and they are only mildly offensive
Edit: I guess I take the view that - like a pirated movie filmed in a cinema - the only people that use/watch it are people that were never going to pay for the real thing anyway