> While it can maintain coherent environments over short sessions, visual details and object permanence can still shift or drift over time—especially as a simulation runs longer, after multiple complex prompt edits, or when navigating throughout the scene. Complex physical interactions, real-time audio accuracy, rendering legible text, spatial accuracy, and long-term consistency are active areas of ongoing research.
Though, I am not sure well this retains temporal stability. I am just thinking about how in 3D rendering, objects are tracked as individual things and only culled, but all the data about them still exists.
Maybe the models need to reason as a 3D space first, then turn chosen objects into renderings?
Or render a low fidelity 3d model for coherence and then raytrace that onto a 2d frame that is then rendered at higher fidelity.
I suspect it is, because the 2D -> 3D models I've seen are terrible, but more importantly the 2D -> 3d models are terrible. Noteworthy because I suspect the number of people who have 3D scans laying around is tiny.
What you're describing is vaguely how video games work, to my understanding. There may be interesting parallels if that becomes a technique (like how distant objects are sometimes still rendered, but at a lower resolution/polygon count).
they got this working decently well for video over a year ago, maybe closer to two years ago at this point, time goes fast. so i assume it'll translate to progress in Genie too
I noticed several inconsistencies in things that stayed in frame. See the wall on the bridge in the Van Gogh example as they walk past it to see the other side.