How do specificity and sensitivity work with lab testing?

Insights and discussion from the cutting edge with reference to journal articles and other research papers.
Post Reply
User avatar
TheBrain
Contributor
Contributor
Posts: 1413
Joined: Fri Oct 03, 2014 12:12 pm

How do specificity and sensitivity work with lab testing?

Post by TheBrain »

I wasn’t sure where to post this, as it’s related to COVID-19, but I believe my question applies more broadly about lab tests.

Below is the article that raises the question for me. I understand that specificity and sensitivity relate to false negatives and false positives. We don’t want either. I’ve read that for a lab test, a specificity above 90% and a sensitivity also above 90% is good, and the higher the better. Also, 100% for either is unrealistic.

Dr. Birx states below that a test that’s only 99% specific “means that when you find a positive, 50 percent of the time [it] will be a real positive, and 50 percent it won’t be.” Is that true or did she misspeak? If it’s true, I don’t know why we do any lab testing.

Can a smarter person that me explain how this works? Thanks for your help.

L.A. County Study: Coronavirus Outbreak Up to 55 Times More Widespread, Less Deadly Than Predicted
https://www.breitbart.com/politics/2020 ... predicted/
Asked about L.A. County study’s findings, Dr. Birx warned reporters on Monday, “These [antibodies] tests are not 100 percent sensitive or specific,” adding:

If you have one percent of your population infected and you have a test that’s only 99 percent specific that means that when you find a positive, 50 percent of the time will be a real positive, and 50 percent it won’t be. And that’s why we’re really asking people to start testing in among the first responders and the health care workers who may have had the greatest exposures because that’s where the tests will be most reliable and then when we have the luxury we can go out to broader and broader communities.
ApoE 4/4 - When I was in 7th grade, my fellow students in history class called me "The Brain" because I had such a memory for detail. I excelled at memorization and aced tests. This childhood memory helps me cope!
User avatar
Tincup
Mod
Mod
Posts: 3558
Joined: Fri Aug 08, 2014 2:57 pm
Location: Front Range, CO

Re: How do specificity and sensitivity work with lab testing?

Post by Tincup »

TheBrain wrote:
Asked about L.A. County study’s findings, Dr. Birx warned reporters on Monday, “These [antibodies] tests are not 100 percent sensitive or specific,” adding:

If you have one percent of your population infected and you have a test that’s only 99 percent specific that means that when you find a positive, 50 percent of the time will be a real positive, and 50 percent it won’t be. And that’s why we’re really asking people to start testing in among the first responders and the health care workers who may have had the greatest exposures because that’s where the tests will be most reliable and then when we have the luxury we can go out to broader and broader communities.
From this

" Sensitivity is the proportion of patients with disease who test positive. In probability notation: P(T+|D+) = TP / (TP+FN).

Specificity is the proportion of patients without disease who test negative. In probability notation: P(T-|D-) = TN / (TN + FP)."

I did the exercise here (see table in link) with this result, and Birx did not misspeak. The issue is the assumption is that 1% are positive AND you are testing everyone. I randomly assumed a grand total of 1,000,000. I used the 99% assumptions.

Code: Select all

Header       Disease_Present     Disease_Absent                 Totals	
Test_Positive        9900        9900                           19800        Positive
Test_Negative        100        980100                         980200        Negative
Totals              10000        990000                      1000000        Grand Total
Tincup
E3,E4
User avatar
slacker
Contributor
Contributor
Posts: 2127
Joined: Wed Aug 03, 2016 6:20 pm
Location: Kentucky

Re: How do specificity and sensitivity work with lab testing?

Post by slacker »

I found Peter Attia's explanation and modifiable spread sheet helpful. The accuracy of lab tests depends on how common the condition is in the community (prevalence).
Slacker
E4/E4
User avatar
TheBrain
Contributor
Contributor
Posts: 1413
Joined: Fri Oct 03, 2014 12:12 pm

Re: How do specificity and sensitivity work with lab testing?

Post by TheBrain »

Tincup wrote: I did the exercise here (see table in link) with this result, and Birx did not misspeak. The issue is the assumption is that 1% are positive AND you are testing everyone. I randomly assumed a grand total of 1,000,000. I used the 99% assumptions.

Code: Select all

Header       Disease_Present     Disease_Absent                 Totals	
Test_Positive        9900        9900                           19800        Positive
Test_Negative        100        980100                         980200        Negative
Totals              10000        990000                      1000000        Grand Total
Thanks, Tincup. I know you're smart and you get this stuff, so I trust what you've done here.

Given that Birx did not misspeak, do you see any point in doing antibody testing? One thing is, I don't actually think we'll ever test everyone (although perhaps that would be ideal). I don't know how much that factors in. In the Santa Clara and LA studies, they tested a small number of the population in each county.

I'm pretty darn sure that my husband had a mild case of COVID-19, and I was either asymptomatic or mildly symptomatic (I felt punky two non-consecutive afternoons before he had symptoms). If I didn't bring COVID-19 to him, I can't imagine I wasn't exposed to it from him. Two days before he had symptoms, we were as close in proximity as two people could be. Let me add that we're acting as if we haven't been exposed because we can't be certain. I've been hanging onto hope of getting antibody testing one of these days to confirm or deny our suspicions.

I know two people who say they are certain they had COVID-19 in December, based on their symptoms only (of course, we didn't have testing yet), and they tested negative for strep and the seasonal flu. They were both really sick but didn't require hospitalization. One lives near me in North Carolina, and the other lives in Colorado. In hindsight, my Colorado friend thinks she should have been hospitalized; she's much better but still recovering.

When I first heard about antibody testing, the interviewee stated that when people test positive for an antibody test, they need to do a coronavirus test to determine if they are currently infected. Which makes sense. The interviewee was from a company that developed an antibody test that hadn't yet been FDA approved, but their self-validation found that the specificity and sensitivity were in the mid-90s.
ApoE 4/4 - When I was in 7th grade, my fellow students in history class called me "The Brain" because I had such a memory for detail. I excelled at memorization and aced tests. This childhood memory helps me cope!
User avatar
TheBrain
Contributor
Contributor
Posts: 1413
Joined: Fri Oct 03, 2014 12:12 pm

Re: How do specificity and sensitivity work with lab testing?

Post by TheBrain »

slacker wrote:I found Peter Attia's explanation and modifiable spread sheet helpful. The accuracy of lab tests depends on how common the condition is in the community (prevalence).
Thanks, Slacker. I didn't see your post until I submitted my response to Tincup. I'll take a look.
ApoE 4/4 - When I was in 7th grade, my fellow students in history class called me "The Brain" because I had such a memory for detail. I excelled at memorization and aced tests. This childhood memory helps me cope!
User avatar
Tincup
Mod
Mod
Posts: 3558
Joined: Fri Aug 08, 2014 2:57 pm
Location: Front Range, CO

Re: How do specificity and sensitivity work with lab testing?

Post by Tincup »

TheBrain wrote: Thanks, Tincup. I know you're smart and you get this stuff, so I trust what you've done here.

As Slacker says, it all depends on prevalence (or assumed prevalence).

In my example, I assumed 1% actually had the disease, so 50/50. If you assume 25% actually have it, then false positives are ~3% and 1% if you assume 50% actually have it.

When I said "you test everybody," I did not mean that literally. If you only tested people who'd had symptoms, you'd likely have a different prevalence and outcome of false positives. In my example, you have 1 million people. With 1% prevalence, you have 10,000 who actually are positive. So you are testing 990,000 who are actually negative. 1% of those will test positive or 9,900. This is the rub! I think in our state, about 10% of the people who get tested (with a swab, not antibody) actually test positive (and only symptomatic people test). If you used that as the prevalence of those who test, then you'd get about 8% as false positives. I understand swabs have issues, too - because you might have the virus, but where you sample doesn't have any.
Tincup
E3,E4
User avatar
Tincup
Mod
Mod
Posts: 3558
Joined: Fri Aug 08, 2014 2:57 pm
Location: Front Range, CO

Re: How do specificity and sensitivity work with lab testing?

Post by Tincup »

One thing to note, with a 1% prevalence, if you test false, the probability that you've not had the bug is very near 100%. In my 1 million case example. There are only 100 false negatives out of nearly 1 million.
Tincup
E3,E4
User avatar
TheBrain
Contributor
Contributor
Posts: 1413
Joined: Fri Oct 03, 2014 12:12 pm

Re: How do specificity and sensitivity work with lab testing?

Post by TheBrain »

TheBrain wrote:
slacker wrote:I found Peter Attia's explanation and modifiable spread sheet helpful. The accuracy of lab tests depends on how common the condition is in the community (prevalence).
Thanks, Slacker. I didn't see your post until I submitted my response to Tincup. I'll take a look.
Great video! Peter Attia explains the matter well. I still need to play with the spreadsheet, as that will help cement things in my brain.
ApoE 4/4 - When I was in 7th grade, my fellow students in history class called me "The Brain" because I had such a memory for detail. I excelled at memorization and aced tests. This childhood memory helps me cope!
User avatar
TheBrain
Contributor
Contributor
Posts: 1413
Joined: Fri Oct 03, 2014 12:12 pm

Re: How do specificity and sensitivity work with lab testing?

Post by TheBrain »

Tincup wrote:One thing to note, with a 1% prevalence, if you test false, the probability that you've not had the bug is very near 100%. In my 1 million case example. There are only 100 false negatives out of nearly 1 million.
Thanks for this and your previous post. My head has finally wrapped this matter.
ApoE 4/4 - When I was in 7th grade, my fellow students in history class called me "The Brain" because I had such a memory for detail. I excelled at memorization and aced tests. This childhood memory helps me cope!
J11
Contributor
Contributor
Posts: 3351
Joined: Sat May 17, 2014 4:04 pm

Re: How do specificity and sensitivity work with lab testing?

Post by J11 »

TheBrain, 99% is a near ideal circumstance.
In many tests the available information is much less precise.

Consider the Receiver Operating Curve (ROC) for APOE genotype in
distinguishing AD from a healthy control. Area under the Curve (AUC)
is 0.90. Here you might have a false positive rate of 25% with a true positive
rate of 90%. This is somewhat surprising as the AD patients had MMSE scores
of 21! They still falsely misclassified 25% as having AD when they didn't?
Then there were the 10% of patients with dementia that were not correctly
classified by APOE.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6805777/
APOE ROC.GIF
You do not have the required permissions to view the files attached to this post.
Post Reply