This seems like a more pertinent alignment issue than guardrails and preventing bioterror etc: does the bot even know what it can’t do? Because the eagerness with which they’ll be wrong about drawing a bounding box concerns me much more than whether they’re willing to talk me through synthesizing LSD

Yeah, I tried to have AI arrange some furniture-shaped rectangles on a black-and-white floor plan. Impossible. I could painstakingly guide it through writing a floodfill to find the corners of a room and parse out the dimensions listed in that room, but at that point it wasn't using its image-seeing abilities at all, and it still couldn't do things like putting a particular rectangle against the bottom left corner.