If I reduced the res a bit, then it worked, so there did seem to be a limit there somewhere. height and width are stored as u32s internally though, so yes, the format itself technically supports it...
I never looked into it more, so might have been a libavif issue, or maybe the way I was using it (maybe it has defaults or profiles?)