undefined

points

[-]

"Imagine being hired as a consultant by the mayor of a fictional city. Your task is to help hire for twenty jobs such as doctors, lawyers, childcare aides,janitors with applicants from four unfamiliar demographic groups: Tufa, Aima, Reku, and Weki. In each round, there is a new job vacancy and four applicants, one from each group, awaiting your decision. Once you make your choice, you learn immediately whether the hire was successful, and move on to the next round. Your goal is to maximize successful hires across 40 rounds, which will be converted into a real bonus compensation"

i remember an tipp our teacher gave us for quizzes: if we need to tick an answer from a b c d. We should choose a letter at random before we start the quiz. With this strategy we maximize our chances of getting more points. The logic is, we minimize the variance of choosing the wrong answer and we should get closer to the expectation value of 25%. Can it be that such a strategy is hardcoded in our brain?

by taurath9 hours ago|

prev|

[-]

The moment code gets written and read back, the decisions made are often treated as gospel by frontier LLMs, even if it was just something that the LLM optimistically created itself. This seems to be one of the core alignment problems to me. See also: Gastown, the agent management project that could only end up working on Gastown, unceremoniously and quietly set aside.

by pjc5022 minutes ago|

parent|

[-]

> Gastown, the agent management project that could only end up working on Gastown, unceremoniously and quietly set aside.

I was ignoring that, but it did seem somewhat intentional by the human running it? There's a lot of "I'm going to use AI to make a better AI-using machine" projects about that aren't really focused on wider application.

by antupis4 hours ago|

parent|

prev|

[-]

Yup and this propagates those clumsy if this_new_code_branch: actual_code_that_matters else: old_legacy_code_that_should_not_be_there

by lelanthran1 hours ago|

parent|

prev|

[-]

> This seems to be one of the core alignment problems to me. See also: Gastown, the agent management project that could only end up working on Gastown, unceremoniously and quietly set aside.

I did not know this! Any link to an announcement or autopsy of sorts (even if not by the initiator of that project)?

I mean, it was pretty expensive, wasn't it? A few tens of thousands of dollars, IIRC?

by palmotea3 hours ago|

prev|

[-]

> ...but the takeaway here seems to be that LLMs are more confident with the initial data that they see and are less likely to chose exploration over exploitation.

You don't say!

"That confirms the real bug: <this obviously totally irrelevant thing that's obviously not the bug, which would take two seconds to disconfirm>."

"You were right to push back..."

by vintermann5 hours ago|

prev|

[-]

Talking about this in terms of exploration/exploitation may be a bit misleading, because from a pure exploration-exploitation perspective, biases wouldn't be a problem if the groups were secretly all identical. If they are, you are "right" to spend zero effort on exploration, your initial inaccurate model that the X are better doctors than Y, will produce no worse results than the completely accurate model.

by freetime24 hours ago|

parent|

[-]

I think this is implied by your comment that the issue is framed improperly, but just to point it out explicitly: the reason that this is a problem is because it would lead to segregation, inequality, and injustice in a society where the biased selection mechanism is used.

Even if it happens to be “optimal” in this case at assigning employees to positions based purely on the information available and their likelihood to succeed, biases can present other issues.

by HappMacDonald1 hours ago|

parent|

[-]

While that may be true in reality no such externality consequences appear to be baked into the game theory of the experiment.

It's like having people or neural networks choose door 1 from door 2 without clear advantage to either of them and without making it clear that one door somehow represents "donating blood" while the other represents "kicking puppies".

by jb19915 hours ago|

parent|

prev|

[-]

Isn’t exploration vs exploitation about the decision-making process, not about the actual reality in the world around you? It doesn’t matter if they are secretly identical or not. The exploration/exploitation trade-off is in the person making those decisions.

by vintermann5 hours ago|

parent|

[-]

I don't understand what you suggest that implies?

by 4 hours ago|

parent|

[-]

deleted

by ben_w4 hours ago|

parent|

prev|

[-]

I think they're saying that while it doesn't matter, the agent and human "do not actutally know" that it does not matter.

Philosophy sometimes says that knowledge is a "justified true belief"*; in this experiment, agents and humans have incorrectly justified a false belief that some applicants are better for certain roles.

* other times, it says this isn't good enough

by Turskarama5 hours ago|

parent|

prev|

[-]

Of course it does, if you start filtering people out at random then you have pointlessly introduced the possibility of randomly filtering out the best candidate.

by vintermann5 hours ago|

parent|

[-]

You have no other information to go by in this scenario, so whatever you do you're equally likely to randomly exclude the best candidate.

by rrr_oh_man3 hours ago|

prev|

[-]

> but the takeaway here seems to be that LLMs are more confident with the initial data that they see and are less likely to chose exploration over exploitation

That’s why I’m of the (slightly contrarian) view that good context management is considerably more bang-for-buck than any type of harness, agent, or other fancy new bandaid of the month.

by joshspankit8 hours ago|

prev|

[-]

Like prompt repetition, I wonder if reminder checkpoints stating ~”question assumptions, stay open minded” would completely remove this problem

by rrr_oh_man3 hours ago|

parent|

[-]

Remember to not think about the pink elephant!

(No, it won’t. At least not while we’re doing self attention)

by jvanderbot7 hours ago|

parent|

prev|

[-]

Have subagent periodically review the work and plan.

by nullc1 hours ago|

prev|

[-]

Now ask the LLM to write a program to perform this task...