Skip to main content

Notebook Medicine

Troubling retractions

(iStock)

Medicine

Troubling retractions

Peer review failures by top science journals may undermine trust in medical science

The Lancet and The New England Journal of Medicine—among the world’s most prestigious science journals—each retracted a headlining study about COVID-19 on Thursday. The implications extend beyond those journals, though, and threaten to erode public trust in medical research at a time when it is most needed.

The Lancet’s study, the better-known of the two, claimed to have analyzed over 96,000 COVID-19 patients from 671 different hospitals. On the strength of this seeming mountain of data, the authors stated that the drugs hydroxychloroquine and chloroquine did not help patients, and that in fact they often harmed patients. The study was published May 22, and its conclusions affected other studies: When researchers learn their projects would harm patients, they are ethically obligated to stop.

Thus, the World Health Organization’s “Solidarity” trial, proceeding in over 100 countries, paused its hydroxychloroquine arm. The ASCOT trial, seeking to recruit 2,500 patients in Australia and Asia, also paused.

But soon, an impromptu online peer-review effort sprang up around the Lancet study, identifying major red flags: For example, the study claimed the prevalence of smoking was almost uniform at about 10 percent worldwide. (World Health Organization data show wide differences, with 13.9 percent of Africans smoking tobacco, 28.7 percent of Europeans, and so forth.) The Lancet study also had only four authors, a number more often associated with small studies at a single institution than with massive studies analyzing tens of thousands of patients. 

Surgisphere Corp., a company founded by study co-author Sapan Desai, claimed to have the data, yet said it was prohibited by “data sharing agreements” from allowing anyone else to evaluate or verify the analysis—an extremely unusual way of handling data for groundbreaking research. Even more basic questions remained unanswered: For one, how could a tiny company persuade hundreds of hospitals on six continents to share patient data?

Hundreds of researchers signed an open letter, posted online May 28, that questioned the Lancet study. On Wednesday the research journal published an “expression of concern” alluding to “important scientific questions [that] have been raised.” Then, the next day, three of the four study co-authors asked the journal to retract the study.

How did it get published at all? Lancet editor Richard Horton has published editorials criticizing President Donald Trump, who has promoted the use of hydroxychloroquine against COVID-19. But even if political leanings influenced his decision to accept the Surgisphere study, politics can’t explain why The New England Journal of Medicine also accepted a recent Surgisphere study about less controversial medications. That study, published online May 1, analyzed whether two common classes of blood-pressure medication increased the health risk in patients with COVID-19 and concluded they do not. This week, as critics questioned Surgisphere’s reliability, the NEJM announced its own “expression of concern.” On Thursday it retracted the article.

Even that was not the end of Surgisphere’s impact: A third journal posted a preprint article online (since deleted, but archived here) relying on Surgisphere’s data and endorsing the use of ivermectin—a treatment for worms and other parasites—against the coronavirus.

All this matters, not simply because the public needs to be able to trust the research behind its medical care, but also because this bad research stopped better research from proceeding. The World Health Organization trial and the ASCOT trial have resumed, but will patients be willing to join? Or will they view the Surgisphere data as reason to stay away? What of the patients in South America now turning to ivermectin on the basis of a study since deleted?

Above all, how did a tiny startup’s hand-waving explanation fool the two most influential medical journals in the world—and a third journal for good measure—at the same time? Did Surgisphere’s talk of “machine learning” and “actionable data insights” dazzle the journal editors into ignoring the red flags? 

Questions outnumber answers right now. Those answers will be crucial to reestablishing public trust in medical research.

Comments

You must be a WORLD Member and logged in to the website to comment.
  • JerryM
    Posted: Fri, 06/05/2020 08:38 pm

    I repeat my comments from The Sift below.  Perhaps this is an encouraging development regarding exposing bias in academia?  High-profile stories of evidence of bias in the soft sciences already exist (e.g. social sciences; see below and https://areomagazine.com/2018/10/02/academic-grievance-studies-and-the-corruption-of-scholarship/).  This appears to be evidence of bias in the hard sciences.

    Posted: Thu, 06/04/2020 08:46 pm

    This is a fascinating development.  Without trying to be conspiratorial, I would include the potential motivation of western researchers or other associated actors to undermine Pres. Trump.  

    I would encourage a read of the following article: https://www.sciencemag.org/news/2020/06/mysterious-company-s-coronavirus-papers-top-medical-journal-may-be-unraveling.   A telling quote: "Chaccour says both NEJM and The Lancet should have scrutinized the provenance of Surgisphere’s data more closely before publishing the studies. “Here we are in the middle of a pandemic with hundreds of thousands of deaths, and the two most prestigious medical journals have failed us,” he says."

    This may bring to mind a retraction in another prestigious journal that involved another politically-charged issue (see https://www.sciencemag.org/news/2015/05/science-retracts-gay-marriage-paper-without-agreement-lead-author-lacour).

  • Cyborg3's picture
    Cyborg3
    Posted: Sat, 06/06/2020 02:01 am

    We were both right on to question world's previous article on this, where it pushed this study.

     
    https://world.wng.org/content/mixed_signals_on_hydroxychloroquine

    It will be interesting to see if Dr Horton will have to eat his words. My gut instincts with the evidence I have heard about is that he will - which Doctors don't like to do. 

    https://world.wng.org/2020/05/a_spoonful_of_coronavirus_medicine

  • RK
    Posted: Sat, 06/06/2020 08:04 pm

    There certainly can be bias everywhere, but that does not we give up on medical research. The haste (especially recently) to post studies, let alone news to media outlets I believe is a good chunk of the problem. The readers of medical journals have provided a long-standing check by analyzing studies, posting letters to the editor, and giving the study authors a chance to respond, but when the journals are at their best, they screen the articles better. The fact that studies in medicine have to be transparent as to how they performed the study and made conclusions helps in analyzing which studies hold up over time. 

    In this day and age where we are tempted to accumalate more info quickly and focus less on quality, it may be wise to read less and meditate (in the Christian sense) more on what we do read. The journals should do the same. 

    One last piece of advice. A few physicians that I have known that have published medical research believed that 80-90% of medical studies don't hold up in the long run. This is not necessarily intent, just that good research is not that common, so do not get swayed by the latest headlines: see if the medical community come together and note the study's quality before adopting what a study says.

  • Cyborg3's picture
    Cyborg3
    Posted: Sun, 06/07/2020 10:55 am

    You seem to miss the fact that this was an attempt to use science as a political tool to attack Trump. It wasn’t just poor research but an example of the “deep state” in action in the scientific community. The main individual behind the scandal was Sapan Desai, an Indian immigrant to the US living in the Chicago suburbs. His company, Surgisphere, supposedly used data analytics to draw scientific conclusions from studies from all over the world.  He showed great bravado in thinking he could use his manipulated data to scuttle the studies on hydroxychloroquine. Here is an interview where you see the guy in action:

    https://m.youtube.com/watch?v=ZVoYZXPUS7w

     Here are two other informative articles on the topic:

    https://www.sciencemag.org/news/2020/06/mysterious-company-s-coronavirus-papers-top-medical-journals-may-be-unraveling

    and

    https://theprint.in/health/star-cardiologist-mehra-data-doctor-desai-story-of-indian-experts-behind-hcq-scandal/436601/

    When I say “deep state” I am referring to a loosely connected group coupled oftentimes by a common political ideology who use their positions to advance that ideology.  The liberal mainstream media puts out the narrative that they are advocating and likeminded ideologues will use their position to advocate it, oftentimes in corrupt and illegal ways. Dr Desai likely picked up the messaging from the liberal news organizations that they wanted to show Trump as dishonest in pushing hydroxychloroquine as a tool in fighting COVID-19. He understood what they wanted and thought he could deliver by using his company’s supposed data analytics capabilities to push the narrative that Trump was deluded and a poor leader by advocating snake oil COVID-19 remedies. I believe Dr. Desai thought he could advance his company by pushing the liberal MSM narrative. 

  • RK
    Posted: Sun, 06/07/2020 12:27 pm

    I did not miss that there will be attempts and have been attempts to inject politics and worldviews to discredit others, and there will be attempts in the future. However, I have been reading medical journals for over 30 years, and these kind of attempts have been the exception than the rule. My pointing out of an eager medical community to review prominent articles for decades is one of the big safeguards to catch what editors miss, and often do. Regardless of if there is a sinister motive behind these articles or not (and typically there is not), research that is not authentic is often uprooted since the methods and numbers do not hold up when evaluated. Trnasparency is a major requirement in publishing medical research, and I was pointing out that most problems in medical research do not have ulterior motives of worldview.  As long as that high ethical standard remains the norm, then the attempt to purposefully falsify, political agenda or not, will be caught by the readers if not the editors. Even then, research that looks sound needs to be repeated elsewhere for validation. 

    My point was that even if the authors of the study had ulterior motives, I am not going to assume that the editors have the same agenda. It does not mean that the editors do not have their own biases in life, but one of the ethos of medical research and journals is to let the numbers speak for themselves. A lot of the failed research does not reach publication, and those who try to make failed research look good get caught. Nobody is perfect nor no system. Even the critics of a study are not always right, since we all have our biases, but the medical research system has been a pretty good self-policing system.

    My concern is looking too hard for political agendas where it is hard to inject it. As noted, the Social Sciences and research based on accepting a certain "norm" of behavior and assuming a certain set of values are more susceptible to worldviews. I would find it hard to believe, though  that some of the people who caught these problems in the hydroxychloroquine research are not "liberal" or "moderates". The hard sciences are harder to induce bias on a permanent bias It is certainly good to be on the lookout for bias and inaccuarcy in the hard sciences, but I think we have to be careful not to be so cynical as to assume evil where it does not exist. The process caught the problems again, like it has in the past. I can see a major plane crash and assume flying isn't safe, when I know that the world has had an entire year where there were no major plane crashes, and that in the big picture, driving is a lot more dangerous. This is certainly concerning, but most medical research is rather "boring" in regards to any worldview.

    All I ask is to look at the big picture. In response, my concern is that we do have a spiritual battle, but to be careful not label good or evil based on political leaning or label certain groups. There are people in the body of Christ who are also moderates and liberals, there are people in the body of Christ who are in the mainstream media. I have seen the conservative media also lie, also purposefully distort the truth in the news. Part of being light is pointing out error and evil, yes, but it is also to not simply hear what we want to hear, not looking for error/bias selectively, not writing certain beliefs or people off. None of us have all of the right answers, and we are all one body of Christ, even if we do not agree on everything. We should sharpen each other as iron sharpens iron, but our political views should not interfere with the love of Christ as one body. 

    One more reminder. C. Everett Koop said that as Surgeon General, that the Reagan Administration asked him to look into research on women's mental state after abortion, wanting to show that women were distressed, depressed after abortion. He looked into the research, found that there was no evidence of that overall. Some women were depressed, some women were happier and no regrets. He hated abortion like I do, but he did not provide the answer that the Administration wanted. He was true to the scientific process. Whatever answers come out for hydroxychloroquine or other things related to coronavirus, let us make sure that the process continues to be open to promote honesty, but also ask yourself if the answer is not what you want, then will you accept it.

  • Eileen
    Posted: Mon, 06/08/2020 07:37 pm

    I know of instances in which doctors are prescribing this for their COVID 19 patients with good results..or planning on prescribing if the test of that individual is positive.  Two of these instances I know about personally.  Also, if the President's doctor thought it would be dangerous for the President, I'm sure he wouldn't have prescribed it for him.  It is amazing to me that the President was chastised for taking a drug that was okayed by his doctor! Obviously, doctors haven't been taking the study seriously either.

  • RK
    Posted: Fri, 06/12/2020 12:14 am

    I am a doctor in primary care. I would not take hydroxychloroquine unless I was in desperate straits. I do not know other doctors writing for it or endorsing it, nor would I write it for others including family.  I have seen studies over the years that looked promising initially seeing a few cases of success, but over time, when enough people got enrolled, the drug proved to not work.The medication has been known to have its dangers before COVID-19 struck, including dangerous heart rhythms, which is another resaon I would not prescribe it. The headlines of this medication has resulted in doctors prescribing the medication to themselves and their families unethically, "just in case" to have a stockpile, and the high demand  has been depriving people who use it for more proven purposes like automimmune disease due to shortages.

    Just because these studies had an agenda of dishonesty does not throw out all of the other studies for COVID-19. There are doctors that do believe in COVID-19, but it has mainly been a research medication for the hospital. Nobody for sure will know about hydroxychloroquine with COVID-19 until the research keeps coming in, and there is enough gray area to argue that the President can take it, hoping the benefit outweighs the risk. However, the medical people I am around. The American College of Physicians, the United States Internal Medicine main organization is not a group I agree with on everything, but on most things. This is their statement: 

    "Due to possibly worse clinical outcomes and known harms, the American College of Physicians advises against use of chloroquine or hydroxychloroquine alone or in combination with azithromycin as treatment for COVID-19, and recommends shared and informed decision making if hospitalized patients are treated with either drug alone or in combination with azithromycin in the context of a clinical trial."

    https://assets.acponline.org/coronavirus/scormcontent/?_ga=2.140520690.1...

  • RK
    Posted: Mon, 06/15/2020 11:03 pm

    This may help give more insight on the studies from fivethirtyeight.com. You can skip the first 1:05 of tangentially related anecdote, then listen to 15:50, but you can listen to the rest if you want. 

    https://podcasts.apple.com/us/podcast/the-case-of-the-bogus-hydroxycholo...