Looking at the Stanford Santa Clara study on COVID-19

A couple people have pointed me towards this interview with Stanford Dr. Jay Bhattacharya about his antibody study of COVID-19 in Santa Clara, CA. His conclusions, which have been widely reported, are that the number of people who've contracted the virus are actually 50x to 80x more than the official case counts, and thus that the actual death rate is down around 0.16% or not much different from the flu.

I listened to the interview and then read up some more on the test to make sure I was understanding right how it worked. Two things that are worth keeping in mind:

1) When they developed their test, the manufacturer of the test did a test for false positives by testing 371 blood samples from before coronavirus was in the US. The test correctly identified 369 of those samples as not having COVID-19, and incorrectly identified 2 as having it. (Dr. Battacharya's team then did their own test on 30 samples they knew couldn't virus antibodies, and all thirty came back as negative, but since that 0/30 result is consistent with the 2/371 result, I don't feel that really changes it.) When they did their actual test, they tested 3330 people and had 50 tests coming back as positive. So if we think about the false positive rate they had when doing their initial check, that means that nearly half of their positive results (18 out of the 50) in the study could have been through the test returning false positives. And we don't know what the range of error rate is. If they tested another 371 known non-COVID exposed samples, would they get another 2 errors, or would they get 4 or 0? This test would be acceptably accurate for testing a population with 10% of 20% of people who actually had coronavirus antibodies, but it creates a huge amount of uncertainty when only 1.5% of the people they tested had it. They did some statistical adjustments to try to deal with that and also to account for the sample bias of which people from the county showed up, but it really makes the whole thing very hard to evaluate. (If you want to read about this in more depth, see the analysis here.)

2) Let's take the test conclusions as being totally accurate and say that the actual infection fatality rate of COVID-19 is down around 0.15%. Does that fit with the other things that we know? Well, as of today New York City (the worst outbreak in the US) has had 10,344 deaths in a population of 8.4 million. That means that the COVID-19 deaths in NYC as a percentage of the total population are 0.12% If the Stanford study is right that the actual fatality rate of COVID-19 is around 0.15%, then we'd have to assume that 82% of New Yorkers have already had the virus. And yet, overall, only about half the people test in NYC are getting positive results. That would mean that among the people who think they have COVID-19 and seek out a test, the rate of infection is significantly lower than among the NYC population as a whole. And that just doesn't make a whole lot of sense. Moreover, it would suggest that some areas such as Bergamo in Italy had an infection rate of over 100% (and that's using only the official deaths, not the "excess death" analysis of mortality which has suggested that Italy probably actually saw 2x to 3x more deaths than the official counts.) I think we'd all like to believe that the outbreak is nearly over in New York City, but there are a whole lot of ways in which the idea that 80% of the population has already had the virus does not pass the smell test.

That hard hit areas simply don't make sense using the infection fatality rates than the Stanford study, combined with the fact that the false positive rate of their tests may contributed nearly half their results, suggests to me that there's something wrong with the overall model. Testing for antibodies to the virus in order to figure out how many people have actually been exposed to it is definitely going to be an important tool for figuring out how widespread the virus has become and what we should do next. However, it looks to me like the Stanford study has probably rushed out with a result with fits a story which many people would like to believe, when the results are in fact pretty uncertain.