As disagreements pile up over easing COVID-19 restrictions and reopening businesses, scientists are racing to gauge how many people have truly been infected. Among other reasons, measuring the pandemic’s spread can clarify how deadly the disease is and how many people are presumably immune to it, which is crucial for anticipating future waves of the outbreak.

The cornerstone of this process is antibody testing, which looks for proteins made by the body’s immune system to combat invading pathogens. Antibodies can identify cases where people have caught COVID-19 even if they didn’t show any symptoms. Early antibody results suggest a substantial portion of cases have been missed by genetic tests for COVID-19, the first line of diagnostics for the disease.

The reported numbers vary widely—21 percent in hard-hit New York Cityversus roughly 6 percent in mostly spared Geneva, Switzerland—which in part is expected. But some of thedifferences are a consequence of rushing to publish results, and in some cases, troubles with the tests themselves.

Experts say that COVID-19 testing has now entered a “Wild West” phase. Inaccurate, unauthorized tests for the coronavirus are flooding the market due emergency deregulation of oversight. Compounding that, flaws in collecting data and analyzing results can produce misleading estimates—many of which would normally be caught by the peer-review process, a safeguard built into academic publishing. But in the mad dash to make sense of this pandemic, many research teams are instead leaning on press releases or unreviewed articles called preprints to spread the word.

“It’s very difficult to really figure out what’s going on,” says Samuel Scarpino, who heads Northeastern University’s Emerging Epidemics Lab. “What we’re facing here are some numbers that are being reported without the context necessary to interpret them.”

One notable example is a recent survey for COVID-19 antibodies among residents of Santa Clara County, California. On April 17, Stanford University researchers reported in a preprint that, of the 3,300 people they’d screened, 1.5 percent were positive. Ultimately, the team claimed that the county actually had 48,000 to 81,000 cases of COVID-19, which seemed implausibly high given only 50 deaths and about 1,500 cases were recorded at the time.

This funky math has been widely criticized, and because the study entered the public sphere without peer-review, it launched a controversial debate over whether COVID-19 is less deadly than epidemiologists have said. Since then, the Stanford team has updated its report, dropping its estimate to a range seen by other studies.

In general, this first wave of antibody testing “ends up being a balance between a quick and dirty approach, and something that’s more rigorous and representative, but will take longer,” says Natalie Dean, a biostatistician at the University of Florida who has spotted problems among other antibody surveys. “So we’re seeing the fast ones coming out the earliest.”

False negative, false positive

A survey’s results are fundamentally limited by the quality of the test used to detect COVID-19.

The problem is, many tests on the market are demonstrably not up to snuff. In the U.K., officials paid $20 million for antibody testing kits from China that didn’t work. The Spanish government had to trash 640,000 antibody testing kits that ended up having an accurate detection rate of just 30 percent. And a recent evaluation of 12 widely used antibody test kits, which declared upfront its funding by Anthem Blue Cross Blue Shield, the Chan Zuckerberg Biohub, and others, concluded that only two had reported accurate results 99 percent of the time—the benchmark for reliability set by the U.S. Food and Drug Administration on May 4.

Prior to this date, the U.S. was flooded with more than 150 unvalidated testsbecause the FDA had allowed companies to sell these products without seeking Emergency Use Authorization. The FDA shifted its policy after Congress opened an investigation into this runaway testing market.