Australian NHMRC under scrutiny
IMPORTANT questions asked in the Senate of the National Health & Medical Research Council (NHMRC) over the 2012-2015 review of the evidence on homeopathy (the Homeopathy Review) have not been accurately or fully answered and in some cases not answered at all.
Senator Stirling Griff put a total of 8 targeted questions to the Department of Health regarding the NHMRC’s conduct of the Homeopathy Review, which is currently under investigation by the Commonwealth Ombudsman.
The Homeopathy Review was the first of 16 natural therapies reviews conducted by the NHMRC, reviews that both the Government and Labor Opposition are now using to justify cutting the Private Health Insurance (PHI) Rebate for natural therapies, restricting healthcare choices.
The NHMRC applied the same ‘Overview’ methodology to all of the reviews under its Natural Therapies Review, an unprecedented method which meant that no original research papers were retrieved or assessed and no subject or research experts were consulted for any of the 16 natural therapies targeted. Further, no targeted cost-effectiveness assessment was carried out for any of the therapies, even though the Department of Health informed the public that this was done, with the reviews being used to justify removal of health fund rebate support for ‘Budget savings’. Available evidence points to use of these therapies producing considerable savings to the healthcare budge.
Answers to Senator Griff’s questions can be found on the Senate Estimates – Questions on Notice webpage (Questions on Notice No. 261, 264, 268, 269, 270, 273, 274, 275).
Below, Your Health Your Choice provides a detailed analysis of the NHMRC’s response to one of Senator Griff’s questions – Question on Notice No. 268 – highlighting significant and concerning inaccuracies. We’ll provide analyses of responses to other questions in due course.
Question on Notice no. 268 – studies with less than 150 participants excluded from Homeopathy Review findings:
Question:
Senator Stirling Griff asked the Department of Health on 30 May 2018—
“The Homeopathy Review created a rule that trials with less than 150 participants were not ‘reliable’, meaning their results did not “warrant further consideration of their findings” (Information Paper, Appendix C) . Does the NHMRC apply this rule to other trials it funds and/or reviews? If so, where else has this rule been applied and what internationally accepted scientific standards were used to authenticate it?”
Facts:
The NHMRC has never before applied a rule that the findings of human clinical trials with less than 150 participants are ‘unreliable’. In fact, the NHMRC regularly funds and collaborates on trials with fewer than 150 participants (see Table 1 below).
The NHMRC has never applied this rule when reviewing any other category of health evidence – it was developed just for the Homeopathy Review. There are no NHMRC or any internationally accepted guidelines or standards that authenticate the rule, and it has never been used by any other government agency or research group in Australia or internationally when assessing research evidence. Then why was it used?
The rule directly dismissed the results of 146 out of the 176 studies the NHMRC initially identified – meaning that these trials’ results were not considered as part of the Review’s findings. A further arbitrary rule was also created – that trials had to have a minimum 100% ‘quality rating’ for their results to be considered ‘reliable’. This dismissed the results of a further 25 trials, leaving just 5 trials deemed to be ‘reliable’.
The Review’s published finding of ‘no reliable evidence’ is therefore based on just these 5 trials – a fact not reported in the NHMRC report.
The N=150 rule was formally adopted by the NHMRC Homeopathy Working Committee (HWC) at its face-to-face meeting on 12 July 2013, four months after the contractor (Optum) had already completed assessing the evidence in mid March 2013. The sample size rule was not specified in the original research protocol that was agreed between Optum, the HWC and the Office of NHMRC on 22 December 2012, which was not published.
Answer provided by NHMRC:
“Studies with less than 150 participants were not excluded from consideration in the evidence assessment. The Homeopathy Working Committee (HWC) deemed that, for the purposes of drafting evidence statements, the outcomes of studies with less than 150 participants were not sufficiently powered to reliably inform conclusions on the effectiveness of homeopathy to treat a particular clinical condition.
NHMRC establishes expert working committees, such as the HWC, under section 39 of the NHMRC Act, to advise NHMRC on the methodology and parameters of evidence evaluation when developing health advice and guidelines. This is in line with international best practice for formulating evidence-based health advice. The HWC has expertise in evidence-based medicine, clinical trials and complementary medicine.
There is no ruling as to the number of participants that is appropriate for a trial that applies for funding by NHMRC. Applications are selected for funding through a competitive peer review process. The grant review panel assesses the application against published criteria. Issues such as trial sample size, experimental design and the power are usually assessed under “scientific quality”. If, in the judgement of the expert assessors, the scientific quality is compromised by inadequate trial size, this will be reflected in the score for that assessment criterion.”
Underlined sections in NHMRC’s response are addressed below:
NHMRC’s answers – Fact Check:
1. “There is no ruling as to the number of participants that is appropriate for a trial that applies for funding by NHMRC”:
Senator Griff’s question was, “Does the NHMRC apply [the N<150 trial sample size] rule to other trials it funds and/or reviews? If so, where else has this rule been applied and what internationally accepted scientific standards were used to authenticate it?”
The question asked was not answered.
To answer the question, NHMRC would have had to explain that it routinely funds and collaborates on trials with less than 150 participants (see Table 1 below) and that there is no such thing as a 150 sample size exclusion threshold to ‘reliably inform conclusions’. If these trials are all deemed to be inherently unreliable simply based on a trial size threshold of 150, why are they being funded by NHMRC?
It would also have had to explain that no other evidence review, whether conducted by NHMRC or another research group in Australia or internationally, has ever imposed a blanket dismissal of the findings of all trials of N<150 as having insufficient participants for its results to be considered ‘reliable’.
Further, it would have to explain that no scientific guidelines or standards or guidelines exist to authenticate the rule.
2. “Studies with less than 150 participants were not excluded from consideration”:
The results of all studies with less than 150 participants were, in fact, completely dismissed from any consideration in the Homeopathy Review’s findings and conclusions.
NHMRC’s response correctly points out that all 176 trials in scope of the Review were described and evaluated in the 300 page Optum Overview Report – but this is not relevant to the question Senator Griff actually asked.
Trials that failed to meet the Review’s arbitrary ‘reliability’ criteria were technically ‘included’ in the sense that their existence as part of the evidence base was described in the Optum Overview Report and they were counted as part of the total number of trials originally identified. However their findings were not described in the Overview Report and their results were not taken into consideration when drafting the evidence statements for each clinical condition, even if the trials showed effectiveness of homeopathy (see NHMRC Information Paper Appendix C, pages 34-35).
Instead evidence statements would simply state, “These studies are of insufficient quality and/or size to warrant further consideration of their findings” and consequently, “there is no reliable evidence that homeopathy is effective.”
In other words, the results of these trials were effectively excluded from the Review’s findings.
In over 900 pages of report documentation, NHMRC did not publicly disclose that the 150 trial sample size and 100% trial quality thresholds together directly dismissed all but 5 of the 176 trials from informing the Review’s finding of ‘no reliable evidence’. Many positive studies rated ‘good’ quality were dismissed from the findings on the basis of the 150 sample size rule.
If the NHMRC reviewed the evidence according to its usual, accepted methods, it would have had to report that of the 176 homeopathy studies included in scope, 88 (50%) reported statistically positive results and only 9 (5%) negative (the rest inconclusive and therefore worthy of further investigation). These findings are strikingly similar to results reported in conventional medical research.
3. “The outcomes of studies with less than 150 participants were not sufficiently powered to reliably inform conclusions”:
This statement is not accurate or truthful, as revealed by documents released under Freedom of Information (FOI) and explained below:
Documents released under FOI reveal that the NHMRC/ Homeopathy Working Committee (HWC) initially created a rule that a trial had to have at least 200 participants for it to be ‘sufficiently powered’. This rule was created in May 2013, two months after the contractor Optum had completed the evidence assessment (NHMRC FOI 2015-16 008 Doc 05 – HWC Teleconference minutes 23 May 2013) and well after the original research protocol had been agreed and finalised in December 2012. Notably the original research protocol, which was never published, did not include any sample size exclusion rule (NHMRC FOI 2014-15-004, Section 52 – Final research protocol for Optum Review, 20 Dec 2012).
FOI documents reveal that on 9 July 2013, NHMRC received independent methodological peer review advice from the Australasian Cochrane Centre (ACC) that challenged the validity of this rule, stating, “It is not clear why a categorisation of adequate and inadequate power was needed (and indeed may be misleading)” and recommending, “It would be preferable not to use the terminology adequate power” (NHMRC FOI 2015-16 008-13 Docs 13 – Australasian Cochrane Centre Methodological Peer Review, 9 July 2013).
In response to the ACC’s expert feedback, on 11-12 July 2013 the NHMRC/ HWC formally abandoned the concept of a trial’s sample size as a measure of whether it was ‘adequately powered’, replacing it instead with a new concept of whether a trial was of “sufficient size to provide meaningful results” (NHMRC FOI 2015-16 008 Doc 06 – HWC face-to-face meeting minutes, 11-12 July 2013). This was now four months after the contractor Optum had completed the evidence assessment and seven months after the research protocol had been agreed and finalised.
At the same time, NHMRC also changed the minimum trial sample size threshold for ‘meaningful results’ from 200 to 150 trial participants. This meant that the results of all trials below this new N=150 threshold were dismissed from consideration in the Review’s findings as not being ‘meaningful’.
The concept of ‘meaningful results’ then underwent further refinement, morphing into the concept of ‘reliable evidence’, which was the final framework used to report the Review’s findings to the public (i.e. ‘no reliable evidence’).
It is particularly noteworthy that none of these midstream manipulations to the research protocol were publicly disclosed in the NHMRC report. Click here for more information about the retrospective, undisclosed changes that were made to the research protocol.
Final word
“The responses provided to the Senate are typical of the degree of obfuscation in the NHMRC report”, said Your Health Your Choice spokesperson Ms Petrina Reichman, who also noted they had been provided under Parliamentary privilege.
“Government guidelines state that public servants are required to fulfil their accountability obligations by providing ‘full and accurate’ information to the Parliament and its committees about the factual and technical background to policies and their administration, governed by The Parliamentary Privileges Act 1987”, she said.
