undefined

points

[-]

How much effort do you put into prompting it? Are you asking it one shot with a simple prompt? Are you giving it to multiple agents and giving them a workspace to collaborate in? You might give it to a trusted colleague who does not know the solution along with an agreement to jointly publish, ask them to go through the process of solving it with agentic support, and then compare your solutions.

Edit: to be clear, I completely believe your experience - just pointing out that it might be interesting for you to actually “try” to solve it thoroughly with the tools, though it is obviously hard to do that in a fair way when you know the solution.

by Routine85029 hours ago|

parent|

[-]

Enough! I think it's a reasonably fair test of the models. Let's put it this way, I use the same process to resolve problems outside my peak expertise, and I find it works well.

I think the solution is easy enough that any competent colleague would come up with it. Some variant on it has probably even been published (hence why I'm not chomping at the bit to publish). I think it comes down to the models being extremely spiky, and the problems they're trying to solve being easily verifiable in code.

by Buttons8408 hours ago|

parent|

[-]

Have you ever told the AI your answer? I bet if you did it would train on it and know it going forward.

by Routine85028 hours ago|

parent|

[-]

Nah. And I have an enterprise plan, so I'm /relatively/ confident that they aren't going to do that.

by eru8 hours ago|

parent|

prev|

[-]

With my personal little test problem, that never made a difference.

However current frontier models got good enough at websearch and thinking through the problem, that they can finally piece together the answer from my own public writing (eg on stackoverflow). And recently someone wrote up an answer to the problem and put it on Arxiv.

by therealdrag09 hours ago|

prev|

[-]

Iirc some of the prior math discoveries involved very very very heavy handed guidance to force the AI to discard prior assumptions, and to try many avenues with MAX effort. So ya models don’t behave like ground breaking theoreticians by default.

by pfdietz7 hours ago|

parent|

[-]

Do a breakthrough!

by eru8 hours ago|

prev|

[-]

Ha, I had a similar little test problem, but the frontier models are now smart enough to read enough online of what I posted here and there to piece together the answer. And someone else has written the paper in the meantime and got it accepted for publishing.

by marshray10 hours ago|

prev|

[-]

Definitely. It will let some smooth-talker with a vibe-coded website raise billions for "Nuclear Fusion AI". Then they can use AI to squeeze your team with lawsuits until you agree to sell cheap, and maybe keep you on as an employee to build it.

by Routine85029 hours ago|

parent|

[-]

Hah. Fusion scammers are a dime a dozen. Unlike SaaS apps or finance, though, you can't sweet talk a neutron into existence, so they fizzle out pretty fast. I'm more worried about fusion dying because nobody comes up with a good enough idea than someone stealing my ideas.

by eru8 hours ago|

parent|

[-]

Going on a tangent: I think fusion is pretty much useless this century. We already have energy generation technology that has high capital costs and low running costs: it's called nuclear fission and works fairly well.

Fission hasn't exactly taken the world by storm.

There are a few nice advantages fusion has, but they are more important once we want to reach the stars as far as I can tell; not this century on this planet.

But I'm just an armchair analyst. You probably have a much better understanding.

by Routine85028 hours ago|

parent|

[-]

Nah I agree with you. Not this century, not this planet. Fusion rockets are really efficient, though ;)

I just find it to be an interesting problem! My hope is it happens before I die. My rule of thumb is anyone who says energy on the grid in < 10 years is a blatant liar.

by tintor6 hours ago|

prev|

[-]

How about swarm of agents controling a fleet of humanoids and automated construction vehicles?

by sheepscreek7 hours ago|

prev|

[-]

I mean, you never know. If we could have simulation models with excellent fidelity, then AI could do all the research and iterations and come up with something very plausible. Cutting down physical world experimental iterations/prototyping.

So effectively, you would spend the first 1000 experiments inside a computer simulation and move to the physical world only when you had enough confidence.

by Routine85027 hours ago|

parent|

[-]

I think you should realize that we already do that. In fact, that's one of the reasons computers were invented in the first place! It turns out that science is about unknown unknowns, and by definition simulations can't answer that.

But agents are really useful for doing some of the really boring stuff, like convergence checks or setting up meshes or whatever.

by sheepscreek4 hours ago|

parent|

[-]

If you're talking about things at the equivalent level of say, finding out if positrons maintain symmetry with electrons for their magnetic moment, then yeah - I hear you. Or if you wanted to research some unknown behaviour of hot plasma at 15 million C, you can't do that in a simulation.

..or can we?

Humour me for a moment. Maybe simulating to the degree that was necessary for cutting edge work was not possible earlier, when humans had to program each and every thing in the system that could affect the result. But with current gen petaflop-on-your-desk level compute + swarms of agents that can work tirelessly to account for every known thing to humanity, maybe the threshold has been pushed far enough to produce some more meaningful discovery work in simulation?