However saying “all tests have error rates, therefore all tests with error rates are equally bad” is a completely wrong takeaway. One of the things Theranos was doing was using traditional equipment to run the tests as a smokescreen for the fact that they didn’t have any working technology. So the results should have been similar to anyone else running them when that was the case.
But there is a problem that when a company is saying they’re doing one thing or using one technology but your actual tests could be run on unknown combinations of unknown lab hardware depending on the day and the whims of a CEO trying to defraud investors, you don’t know what kind of results you’re going to get.
Real labs have established controls and ship reference ranges for their test results because they control their processes. If Theranos was doing who knows what to keep the test results flowing to keep the illusion up for the primary purpose of satisfying investors, that’s not what you want to see out of your test lab.
> Google tells me the FIT test for cancer has a false negative rate of between 8% and 26%. That appears to be likely worse than the Theranos numbers you cited here.
You’re looking at the wrong thing. The Theranos number above is for tests they determined had to be voided because they weren’t run properly due to procedural errors they identified. Their own incompetence or fraud. And those are just the ones they admitted to!
The false negative rate for the test you Googled is a false negative rate, which is a simple fact that tests can’t reliably diagnose everything even when run properly. Tests look for markers in samples as a proxy for a condition, and those markers aren’t always present at levels that meet the thresholds even if the condition exists. Even when run absolutely perfectly, a test like that will have false negative rates.
The part you’re missing is that the Theranos voided and invalidated tests were due to problems on top of the normal test variation rates because they were from Theranos doing things wrong, which is directly downstream of Holmes manipulating investors.
> But there is a problem that when a company is saying they’re doing one thing or using one technology but your actual tests could be run on unknown combinations of unknown lab hardware depending on the day and the whims of a CEO trying to defraud investors, you don’t know what kind of results you’re going to get.
Exactly. They were using Siemens machines, but they often had insufficient blood because they were trying to maintain a pretense of "nano collection". Siemens' machines weren't even trying to pretend they could work like this, so Theranos kept trying to figure out ways to dilute or diffuse the samples they did have to get to a usable volume for the tests. But of course every experimentation there was destructive, so they had even less to use.
So we're not talking about error rates on properly calibrated machines, with sufficient blood that re-tests could be performed before needing another collection, we're talking about error rates on effectively misusing machines.
Also, the standard for tests in the medical world is usually much higher than 11%, except for things that are life threatening for which no better test exists.