upvote
Interesting to see the "io_uring vs io pressure measurement" issue come up here; it's turned up on qemu-devel recently too: https://lore.kernel.org/qemu-devel/4EF13FE7-E6F3-42E2-83F9-6...

If I am reading the thread correctly, the io_uring maintainer's position is that applications trying to use the metric that "poll via io_uring" handles differently from "poll via epoll or select" are fundamentally misguided and measuring something that doesn't make sense, so they don't want to slow down io_uring to placate those apps.

reply
From https://lore.kernel.org/qemu-devel/962d44d3-ca7b-4cc8-b0f7-7...

> iowait is fundamentally incompatible with sane accounting in the first place, that is the problem. It really tells you nothing, and was deprecated since the days of uniprocessor systems. The problem here is not how iowait is done in the kernel, it's that people are using this metric in the first place. Which is why you just want to NOT do that, and do sane accounting on your disk drives, if you care about that kind of thing.

> Which is why I refuse to cater to any kind of iowait accounting metrics, it's just adding overhead to support a metric that is largely nonsensical in the first place. It doesn't answer any questions that an application/tier owner would want to know. Those questions might be things like "am I I/O bound?" or "is the device I'm using being slow?". I wish people would realize that, rather than continue to place value in iowait as a metric for ANYTHING, and stop trying to work around it.

It sounds more like their position is "iowait is a useless metric" and that polling with io_uring completely breaks iowait metrics but fixing it would add a minor slowdown so they refuse to do so because they don't think it's a valuable metric to begin with.

reply
Is there a good replacement for iowait?
reply
An app can use io_uring_set_iowait to change that behavior. Since a terminal’s waiting on I/O isn’t hugging the disk but just waiting on mouse and keyboard input, that seems like the honest thing to do.
reply
Amazing. How did you figure this out?!
reply
I asked Claude to help me debug and research it.

It looked at fossilize code (open source component used in the process), found a couple of threads online that suggested io_uring/iowait and told me first to try closing the ghostty windows and then to try to switch to epoll. This did the trick.

I have a very vague understanding of what's going on (actually better now that I've look at the links/resources from the replies in this thread!), and would probably have been able to figure it by googling in my own. But, my first hunch was that it's a problem with steam, proton, or the game, so I'd probably waste a lot of time in that direction first.

reply
I hate this. We have a whole operating system doing its absolute best to prioritize hardware resources. Don't be out here throttling yourself.

Especially for something like compiling shaders. It literally never happens unless the user is about to go play a game, which is going to consume most resources anyway. To the user, there's nothing more important running right now.

reply
My motherboard crashes when I peg all the cores. Presumably it can't maintain the core voltage under load. Vendors need to guard against these scenarios by being extra cautious because it's a hard problem to diagnose that they'll get blamed for.
reply
Guarding against that seems like a bad idea, but even then just use a couple less threads than physical cores. Don't get too clever.

You in particular should probably disable a couple cores until your hardware stops being defective. Have you looked into updates and voltage droop compensation settings?

reply
That sounds like a faulty motherboard. Return it.
reply
Users don't care that it's a problem with the motherboard. They have a working computer, they buy a game and it doesn't work, they blame and return the game.

Users can be unreasonable, but in this case I agree - if I reported an issue to a vendor and their response was "return your motherboard" I wouldn't work with that vendor again.

reply
We can't form the world around literal bad hardware. That's completely insane. The world would fall apart.

> I wouldn't work with that vendor again.

Possibly rightly so, if it's bad motherboard hardware. Could also be a bad power supply, bad ram, bad CPU, user misconfiguration, etc. It's a complex system, where everything has to NOT be bad for it to be deterministic.

There's a necessary level of competence required with self built systems, or any other task in human life that you're not paying someone else to do for you. That's fine. Go buy a MacBook or a pre built system if that threshold of competence can't be met. That's why they exist, for people that can't or don't want to bother.

reply
Haha, that really depends. If you look at it from the viewpoint of a failsafe, like brakes in lifts that activate without central control, backup parachutes, etc. Then it makes some sense for resource intensive software to guard against weird edge cases like this. It's just that they measured the wrong thing.
reply
Those have nothing to do with the required assumption of determinism in the hardware that software must enjoy.

You're examples are flawed. Those are to safely halt operation on failure. What you're suggesting with the software is to limp along pretending everything is ok.

The proper test, more in line with your others, would be to catch failure and halt operation. Since, in your case c the human isn't competent enough for that, it would maybe be a self test initiated by the operating system, then a refusal to continue when the hardware problem is identified, emulating all the periodic inspections that are part of all the proper operation of all the physical systems your mentioned.

reply
We don't use elevator brakes or backup parachutes as an excuse to let the primary fail under normal load. Those are for emergencies.

I think the elevator analogy to this bug is that the elevator goes too fast and triggers the brakes if you input target floors more than 8 apart. You don't fix that by having the user or user software be more careful about button placement, you fix the hardware/firmware.

reply
Sounds like they don't have a working computer. Computer users occasionally do things that end up pegging all cores, so I'm pretty sure that someone in that case would know their computer has some kind of issue.
reply
Why aren't you storing all your variables with reed solomon encoding in case you have a user with bad ram?
reply
That seems like a you problem - undervolt your cpu.
reply
It’s just reality that motherboards can malfunction or degrade in peculiar ways that a manufacturer’s quick boot up checks cant always catch. No matter the reliability claims.

If they want to cater to customers who foolishly reject reality and take every claim from manufacturer’s at face value… then foolishly misaimed blaming seems expected?

But if they just roll over and swallow the foolish blaming without even challenging customers to double check, then they are also facilitating more nonsense.

reply
But why use that metric. The OS already has support for this.

Why not just re-nice the process or whatever equivalent for the threads that do the compiling? Tell the kernel what’s going on, it will handle it for you.

Don’t hack your own solution.

reply
Especially on Linux, the default throttling just doesn't work very well.
reply
> To the user, there's nothing more important running right now.

Well, to a certain degree. I have a relatively uncommon GPU, and run games on Steam (and Linux), so literally every game does shader compilation for me. Even with a beefy CPU and beefy GPU, if the shader compilation uses more than 16 threads, my computer basically freezes until its done with it, as it's so performance intensive, kind of have to throttle things if I want to be able to use the desktop for the 10 minutes the shader compilation takes sometimes.

Edit: "Do you use Linux? That only happened to me when I was using a dram-less ssd for swap and doing IO heavy operations on the NFS mounted drive"

I do! I do have NFS drives, but not for Steam games or anything else relevant. I'm also not using swap at all. RAM seems fine during this, it's just pegging all cores to 100% usage immediately and stays there, making everything else suffer.

reply
> Even with a beefy CPU and beefy GPU, if the shader compilation uses more than 16 threads, my computer basically freezes until its done with it

Strange. This never happens to me on Windows. I only notice that the fans spin up a little because all cores are pegged.

reply
i've only seen the shader compilation on linux.
reply
Do you use Linux? That only happened to me when I was using a dram-less ssd for swap and doing IO heavy operations on the NFS mounted drive, since the system will use ram as a buffer (which pushes things to swap)
reply
My steam machine is netboot diskless nfs only.(probably a mistake, but I have a nice NAS and so leaned hard into it) However, having no swap worries me. for openbsd they document how to swap to nfs. But the linux docs say not to do that. What is the best practice here? swap to an iscsi block?
reply
> Even with a beefy CPU and beefy GPU, if the shader compilation uses more than 16 threads, my computer basically freezes

Sounds like a scheduling problem that should be adressed at the OS level.

reply
Seems a deep one if so. I've tried a bunch of alternative schedulers to try to avoid the issue, like Bore and BMQ, but seems to happen regardless. Been like this on previous computers too, fwiw, unless I turn down the concurrency for shader compilation stuff will lag heavily until it finishes.
reply
> tried a bunch of alternative schedulers

1. Have you tried enabling full preemption as well? I run Debian, and before I did that I had lots of stutters from any kind of background load thanks to dynamic preemption.

2. Even then, ZFS was causing stutters until I lowered it's priority as well (spl_taskq_thread_priority=0)

3. On non-ZFS drives, try setting your I/O scheduler to Kyber for NVME or BFQ for everything else. "mq-deadline" isn't great for interactive use on a SATA SSD, even for non-boot drives, and even NVME sometimes stutters under "none" for some people.

> I'm also not using swap at all.

4. zswap (combined with zstd compression, "min_ttl_ms=1000", and "vm.page-cluster=0") and "defer+madvise" memory compaction can help. (Something about the kernel really liking having swap on-hand.)

Steam runs shader compilations at batch policy and nice 19, so without RAM or I/O bottlnecks even gaming shouldn't be too bad (though from personal experience it still induces some stuttering). It's just most distros are tuned horribly for this. (Not to sound flippant, I understand the frustration of "just try this" and it doesn't work, but it's everything I can think of RN.)

reply
You could try cpu pinning or ulimits to limit the program from using one core so there's always one free.

But it's also possible that the problem is in the kernel and the shaders are actually doing some system call that's locking up the kernel. Like maybe there's a bug in your cuda or nvidia driver that's causing the kernel to lock.

reply
io_uring you say… one of the most infamous subsystems for security, best reserved for stringent performance requirements. Why on earth would Ghostty default to that?
reply
Maybe because one of ghostty's selling points is performance.

Also if you're running untrusted code that attacks your not-very-common terminal emulator, you're probably in more danger from the "untrusted code" part. Threat model and all that stuff.

reply