Skip to content

About that Hydroxychloroquine for COVID-19 trial

HydroxychloroquineIn the past days there was some hype about an old Malaria drug, Chloroquine and the closely related Hydroxychloroquine, and whether it might help against Covid-19. Particularly Donald Trump indicated he thinks the drug is promising and Elon Musk tweeted about it.

Hydroxychloroquine showed promising results in an in vitro study, but that is obviously no evidence that it will do any good in humans. While more trials are running, the only evidence in humans as of today is a study from a hospital in Marseille, France, but that study is riddled with problems. The trial is still ongoing, but preliminary results have been published in a Preprint on Medrxiv, on the hospital’s webpage itself and in the International Journal of Antimicrobial Agents.

While I understand the basics of a clinical trial, I’d like to point out that I am no medical scientists. Most of the issues here were spotted by other people and I have linked to the sources.

Not randomized

The most obvious issue with this study is that it is not a randomized trial. The study has a control group to which the treatment is compared, but the patients were not randomly assigned. The control group consists of patients from a different hospital and patients who refused the treatment or had other exclusion criteria.

This alone is extremely problematic, because such a selection can introduce all kinds of differences between the two groups that impact the outcome. For example there’s an age difference of more than 10 years between the two groups.

It is therefore possible that the treatment group got better due to other factors.

Surrogate Endpoint

The main outcome the study is measuring is the viral load in patients. This is an example of what is called a surrogate endpoint in clinical studies.

Surrogate endpoints are things measured that are believed to be indicative of a patient’s disease and how bad it is. However they are not what patients really care about. In the case of Covid-19 the most obvious parameter to care about is whether patients survive.

The reason a surrogate endpoint was used here is relatively obvious: It allows faster results, but it has the downside that the results may be less relevant. It is very well possible that a patient with a higher viral load, but mild symptoms may be better off than a patient with a high viral load who needs to be transferred to an intensive care unit.

In a podcast German virologist Christian Drosten discussed this study and raises the suspicion that due to the significant differences of the treatment and control group and the measured viral load it may be that the patients in the treatment group were in a later stage of the disease, which explains the perceived effect. (The podcast is in German, here's the text translated via Google translate.)

But whether that suspicion is true or not: The lack of randomization makes it basically impossible to know whether the measured results were due to the drug or due to patient selection.

Exclusion of Patients

In a comment on PubPeer Elisabeth Bik raises another severe issue. From 26 patients in the group that got the medication six were excluded, because they stopped the treatment. From these six patients three were transferred to an intensive care unit and one died. In other words: The excluded patients didn’t do particularly well.

If patients are excluded from a study this can obviously cause a bias. If a study excludes patients doing particularly bad on the drug this will cause the medication to look better than it actually is.

This effect is well known and the usual way to counter this is to do an intention to treat analysis. What sounds very technical simply means that you collect the results of all patients that you assigned to the medication at the beginning – no matter if you continue the treatment or not. The results of this study contain no intention to treat analysis.

Inconsistency between Trials Registry and reported Result

The main result published by the authors is the viral load of the patients on day 6 of the treatment. The study was preregistered in the EU Clinical Trials Register.

In the preregistration it says that the viral load end point will be evaluated on Day 1, Day 4, Day 7 and Day 14 of the trial. However the main result published is the viral load on Day 6. This is a form of so-called outcome switching.

On the Medrxiv page the authors write that they have followed the reporting guidelines of the EQUATOR project. EQUATOR is an initiative to improve problems with trial reporting and provides checklists for scientists so they follow proper reporting procedures. Yet while the authors claim they have "uploaded the relevant EQUATOR Network research reporting checklist", that is not true. The checklist is not available on Medrxiv.

On Twitter Gaetan Burgio points out another problem: For most of the patients in the control group the viral load was not measured. In the table of the results for many patients only a “positive” viral load is reported, but no exact number. This means the main endpoint of the study was not measured in the majority of the control group.

Another commenter on Pubpeer, David Norris, pointed out that some patients in the table were reported as negative, but had a viral load afterwards. This would indicate that the tests were reporting false negatives – patients that look like they were healthy in a test, but continued to be affected by the virus later.

The paper was investigating Hydroxychloroquine, but it also makes a claim about the chemically similar Chloroquine: “[…] an early clinical trial conducted in COVID-19 Chinese patients, showed that chloroquine had a significant effect, both in terms of clinical outcome and viral clearance, when comparing to controls groups”.

If you follow the cited reference however there is no such trial to be found. The linked publication is a very short PDF making claims about clinical trials being performed in China and it contains a list of references to the Chinese clinical trials registry. If you try to follow up on them you won’t find any results, just trials registry entries about trials that are, if you believe the trials registry, still recruiting patients.

This is an incomplete list of issues raised about this study. Further discussion of various issues with this study can be found in the comment sections of Merdxiv and PubPeer.

There is currently a lot of media hype about Chloroquine and Hydroxychloroquine. Given the current pandemic it is understandable that many people are desperately looking for positive news and potential treatment options. But the treatment options should actually help patients, and only proper scientific studies – and that usually means randomized controlled trials - can show that. Right now we don’t know if Chloroquine or Hydroxychloroquine have any benefit. I don't see how doing poorly performed studys that tell us nothing is of any help.

There are other ongoing studies for these drugs and hopefully some of them will be performed with proper methodology, so we will know better if they have any benefit for Covid-19 patients. Though it’s good to remind oneself that there are far more medications that show promising results in pre-clinical studies than there are medications that actually work. It is a maybe frustrating, yet rational default assumption that most potential drugs don’t work.

Image source: PubChem Identifier: CID 3652

Trackbacks on : PingBack

Unfortunately, the contents of this trackback can not be displayed.


Display comments as Linear | Threaded

tgafpc on :

And the Australians? There's more evidence than just that one study.
50% efficacy is game changing in this case. Just saying.

F30 on :

@tgafpc: What do you mean by "the Australians"? This here?

They have started a clinical trial, after "some patients" have "completely recovered without any trace of the virus left in their system". Which, as they admit themselves, most patients do after all.

I'd love to be proven wrong, but I've now followed a lot of claims that there was strong evidence. And I couldn't find any published clinical study, besides the weak one discussed here.

andrew korf on :

There is trial being don the right way on the efficacy of Thalidomide for treating Covid19. Looks like results are due by the end of the week... there's a reason why we do things the right way.

gocorona on :

what about Belgium and Netherlands using it as preferred treatment?

F30 on :

@gocorona: These national guidelines do not incorporate magical knowledge nobody else has available. The only thing I know about the Dutch guidelines is what is cited on page 9 of the Belgian document you linked to, so I'm just gonna discuss the Belgian ones.

On pages 2 and 3 (for the record, we're talking about version 4 here), they refer to the in-vitro studies (footnotes 6 and 14), an ongoing Chinese clinical trial (footnote 12) and the clinical study discussed right here (footnote 15). They mention "Substantial reduction of […] viral load […] compared to placebo", which I think is a false interpretation of that study. The Chloroquine paragraph on page 4 also explains that the recommendation is based on in-vitro studies and the French clinical study discussed here.

amc on :

China has been publishing results since February that Chloroquine has been successful against COVID-19.

Letter Referencing Chinese Recommendations, Studies, and Papers:

Chinese Paper:

Paper Showing Chloroquine was effective against SARS (an extremely similar virus, this virus is called SARS-COV-2 because of it's similarities)

Anyways, there is much more evidence then one preliminary French study that Chloroquine and related drugs will help against COVID-19 (SARS-COV-2).

Hanno Böck on :

I discussed the first two papers you refer to in the blogpost.

One doesn't contain any evidence, just a list of trials registry entries. The other is in vitro evidence, it means that the drug works in a petri dish, not in a human.

The in vitro evidence is a justification for further research, but certainly nothing more. Unfinished or unpublished trials trials aren't evidence for anything.

Jerome M Taylor on :

Are you a member of the nattering nabobs of negativism??

John Doe on :

There were several new trials started the beginning of this week, one of them in France. Apparently it got good results:

Hanno Böck on :

There's only one new trial that has been published in the meantime, it's from china and if anything, it indicates that hydroxychloroquine doesn't work.

Raoult has published another paper, but that's even less meaningful than the first one, it doesn't have a control group at all. That's not a clinical trial.

John Doe on :

Not true I think:
1. There were 2 published chinese studies: one of them with 30 participants (15 treatment + 15 controll) which indicated that HCQ doesn't work (they had 400mg per day, while the French had 600mg) and another one with 62 participants (31+31) indicating that HCQ helps.
2. The second French trial is much better - it has 80 participants. No control group but in statistics you can have two sample tests and one sample tests. In a one sample test you would test in general some kind of goodness of fit wrt some baseline distribution, which we obviously have in the face of more than 1 million detected infections worldwide. I understand their objection against the controll group version of the test - it is cruel to give people a fake pill in the midst of a severe pandemic. Out of these 80 participants they had more than 50% with preconditions like diabetes, hypertension etc.
Summing it up - focusing on only two sample tests is in my view dogmating in the current situation. One can also work with one sample goodness of fit tests
3. Indirect indicator: on you can see several large trials starting which are about to test the efficacy of HCQ regarding the covid-19. These are expesive undertakings - do you think they will be started if there was no good indication they would produce good results


Add Comment

Enclosing asterisks marks text as bold (*word*), underscore are made via _word_.
Standard emoticons like :-) and ;-) are converted to images.
E-Mail addresses will not be displayed and will only be used for E-Mail notifications.

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.

Form options