I think I'm gonna have to do it anyway, because some players claim they get nausea when playing at such low resolution (320x240), and the only way to give them higher resolutions that perform reasonably is to have it hardware accelerated.
Renderer is abstracted away already, but the real difference would probably be occlusion culling... With raycasting, I get it for free, but if I'd go down the hardware accelerated path I'd have to pick something more clever.
Raycasting and software rendering in general tends to scale poorly with resolution, even with vectorization and all the bells and whistles of modern CPUs.
As an example this[0] video shows the benchmark from Post Apocalyptic Petra running on my previous GPU (RX 5700 XT) which all it does is build a per-frame (client-side) vertex-buffer in OpenGL 1.1 (the engine was made for actual retro PCs running DOS and Win9x so it does some rudimentary occlusion culling but that mainly affects 90s hardware, not anything released since 2000 or so). If anything, the rendering has so little overhead that half of the framerate is "eaten" by the FPS counter overlay :-P.
Thinking about modern games, a single character model probably has more vertices than my entire level (and yours probably), so it's definitely reasonable to expect occlusion culling for such simple geometry might actually reduce performance rather than increase it.
Meanwhile more current games have much higher polycounts, easily going above 100K triangles - e.g. Dante from DMC5, a ~7 year old game, apparently has ~190K triangles and that had to run on the more anemic PS4/XBone hardware :-P (though i'm not sure if it used the full 190K model there or some cut down version).
[0] http://runtimeterror.com/pages/iv/images/1073c7062db40837240...
But ignoring the GPU you have on your system is boring