Is This New Treatment Any Good?

Authored by Dr Naomi Hartree, 20 Apr 2011

This article is for Medical Professionals

Professional Reference articles are written by UK doctors and are based on research evidence, UK and European Guidelines. They are designed for health professionals to use. You may find one of our health articles more useful.

This page has been archived. It has not been updated since 20/04/2011. External links and references may no longer work.

When a new drug or treatment is launched, we have to ask if it is one that we should use. Does it represent a real advance? Be aware that with new drugs, a great deal of money has already been invested in the product, and the marketing team wishes to present it in the most favourable light to enhance sales.[1]

Amongst the questions to be answered are:

  • Is it relevant to my patients?
  • Is this drug a new concept, or a variation on a theme? Perhaps it is a "me too" drug in a lucrative but highly packed niche such as another angiotensin-converting (ACE) inhibitor.
  • How much is known about its safety?
  • Does it represent a significant advance on current options? Consider the following:
    • Is it more effective?
    • Has it fewer adverse effects?
    • Is it cheaper?
    • Is the dosage or route of administration more manageable?
  • How good is the evidence about it, and is any information missing?
  • Do those promoting it have a conflict of interest?
  • Are the benefits worth the risk of adverse effects (known and unknown)?
  • Would I take this treatment myself?

The accepted principle is that clinicians should "trade a known risk for an unknown risk only when there is a reasonable expectation that the new therapy is better."[2]


New medicines in the UK require a licence from the Medicines and Healthcare products Regulatory Agency (MHRA).[3] They have undergone clinical trials, but this does not mean that the product is safe - for example, if the trials or follow-up are inadequate.[4] The number of patients tested in early trials is unlikely to be sufficient to ensure that all adverse effects are discovered. For example: the diabetes drug troglitazone was withdrawn 3 months after it was released because of hepatic side-effects.[2] Later, rosiglitazone was discovered to have important cardiovascular side-effects. Some argue that it was licensed too soon, and that the public is not well served by the current system of drug development.[5, 2]

Safety monitoring

Licensing decisions take into account the seriousness of the condition being treated. For example, more uncertainties or adverse effects will be acceptable for an anticancer drug than for a painkiller used in minor illness.

New drugs carry a "black triangle", indicating that they are "intensively monitored medicines". Monitoring is continued until the MHRA is satisfied that the drug works safely in large numbers of people.[3]

For any black triangle medication, report ALL adverse reactions:[6]
  • Include non-serious reactions.
  • Even if not certain whether they are due to the drug.

Nondrug treatments

  • Nondrug treatments, such as surgical procedures, psychological or social interventions, and complementary medicine, also require evaluation. However, with nondrug treatments it may not be feasible to use blinding in trial protocols.
  • Some treatments may have a net harmful effect. For example, in post-traumatic stress disorder, negative outcomes were reported for some types of critical incident debriefing.[7]

Informed consent[8]

Good medical practice requires that doctors should tell patients about:

  • Options for treatment, including the option not to treat and uncertainties.
  • Details of the treatment.
  • Adverse effects:
    • Common adverse effects.
    • Serious adverse effects - even if rare.
    This information should be shared in a way that the patient can understand.

Newly developed drugs and treatments have not usually been tested on pregnant or breast-feeding women or children, so there will be a lack of dosing and safety information for these groups, and indeed the treatment may not be licensed for them.[9] Older, more established treatments are the first choice. If a new treatment seems necessary, consult an expert, the National Teratology Information Service (for pregnancy)[10] or the British National Formulary for Children (BNFC) - for children and breast-feeding.[11]

The authors and setting

  • Who is presenting the evidence; do they have competing interests? Research suggests that industry sponsorship is widespread and is associated with pro-industry conclusions.[14, 15]
  • Is this a reputable journal or conference? Results from a recent study suggest that pharmaceutical advertising has an influence on the content of free medical publications (those that rely on advertising for revenue).[16]
  • Is there peer review?
  • Who are the authors? Bear in mind that "ghost writing" occurs,[17] and that "independent" research companies may also be vulnerable to commercial pressure.[18]

Study design

What type of study was done? The "gold standard" for measuring the effectiveness of a drug is the randomised controlled study (RCT).
This type of study may be:

  • Single blind - the patient does not know which medication is being taken.
  • Double blind - neither patient nor researcher knows which medication is being taken.
  • Double blind, crossover - the patient will be taking active compound A and a placebo B, or new compound A and existing compound B, and then cross over to receive the other compound or placebo. Neither the patient nor the researcher will be aware which compound the patient is taking at any one time, and the subject is, in effect, his own control.

Look at the methods used section:

  • Is the design biased in terms of patient selection, inclusion or withdrawal?
  • Was there true randomisation of the patients?
  • Is the study adequately powered? This means, is there a large enough number of subjects in the study? It is possible to calculate the number required before embarking on the study. The person reading the paper is unlikely to check such calculations but will simply ask the question, "Do the numbers look reasonable or are they rather small?".
  • What outcome (end point) is used? Be cautious about:
    • Surrogate outcome measures, eg the use of lipid values as an outcome, rather than actual clinical events such as myocardial infarction. If surrogate end points are used, there should be good evidence that they are valid for the condition being studied.
    • Composite end points, eg combining non-fatal myocardial infarction and death. This can make a treatment appear more effective, but the two outcomes are clinically different and are best considered separately.

Validity of results

When assessing the validity of the results it is important to answer several questions:

  • Were the groups made up of similar patients? Were the groups treated equally in all ways except the intervention?
  • Follow-up - were all the patients entered into the trial accounted for at the end? NB: if many patients were "lost to follow-up", be wary of the study's results.[4] The "lost" patients may have had adverse effects or less benefit from the treatment.
  • Have withdrawals and adverse events been recorded for both treatment groups?
  • Were the patients analysed in the groups to which they were randomised? This is an important concept. For example, if a cancer trial puts patients who were too ill to receive the active intervention into the control group, this unfairly favours the intervention group.

Interpreting the results

Are the results clinically significant?

  • It is important to consider what the new treatment was compared to - for example, to a placebo or to an existing treatment.[2] Is the comparison in the trial relevant to your patients?
  • For example, a trial of a hypotensive agent may be able to demonstrate a statistically significant fall in blood pressure, but that fall may be so small that the clinical benefit is negligible.

Are the results expressed in a meaningful way?[19, 20]

Results are often quoted as relative risk reduction, which makes the benefit of the new treatment appear more dramatic. However, absolute risk reduction and the number needed to treat (NNT) are more useful in clinical practice, because they take into account how common the outcomes are, and therefore the size of the treatment effect in practice.

The NNT is calculated as the reciprocal of the absolute risk reduction, i.e:
NNT = 1/absolute risk reduction as a fraction.

Similarly, adverse outcomes of a new treatment can be expressed as the number needed to harm (NNH). This can then be combined with the NNT to give the likelihood of being helped or harmed (LHH). The LHH is calculated as the ratio of NNT to NNH, ie LHH = 1/NNT:1/NNH.

Note that absolute risk measures, including the NNT, only apply to the population studied (or a population with similar baseline risks). And, as with all statistical estimates, the estimated values should be accompanied by a confidence interval.


Here are the results of a hypothetical trial of a treatment (T) for a condition (C):

Example: trial of treatment T for condition C
Results of trial (% patients)Deaths from condition CSurvival from condition C
No treatment20%80%
Given treatment T10%90%

These results can be expressed in various ways:

  • Treatment T halved the rate of death from C (relative risk reduction).
  • Treatment T reduced deaths from C by 10%, or increased the survival rate from 80% to 90% (absolute risk reduction).
  • For 10 people treated with T, one death from C will be avoided (NNT).

Significance and confidence intervals

Statistical significance is often taken as P≤0.05, meaning there is a 1 in 20 probability that the difference between the two groups was due purely to chance.

Confidence intervals: these express the degree of uncertainly about the "true" result.

For example, suppose two trials appear, superficially, to have identical results. One is a small trial comparing A with B and the respective scores were 40% and 50%. Calculation of confidence intervals on the small sample show that there is a 95% probability that the "true" result lies somewhere between ±7% for each group. This means that there is a 95% chance that the "true" result for A is between 33% and 47% and the "true" result for B is between 43% and 57%. There is also a 5% chance that the actual figure lies further out. In this case, the result for A may be as high as 47% and the result for B may be as low as 43%. Hence, there is overlap between the confidence intervals and so the difference between A and B is unproven. Another trial has apparently similar results with A and B at 40% and 50% respectively but, being larger, the confidence interval is only ±3%. This means that there is a 95% chance that A lies between 37% and 43% while B lies between 47% and 53%. There is no overlap and so the difference between A and B has been demonstrated.

Example using risk values, NNT and cost

A hypothetical scenario: Mrs Z requests your help to encourage the PCT to fund a treatment for her cancer. She says that it halves her chance of dying and adds that £20,000 does not seem a high price to save a life. You find that the cost of treatment is actually £20,000 a year and the duration of treatment is usually 2 or 3 years. This makes the cost of a course of treatment £40,000 or £60,000. Let us call it £50,000. The risk of recurrence of her cancer is reduced from 20% to 10%. On this basis the NNT to prevent one recurrence is 10. This makes the cost of preventing one recurrence £500,000. Not everyone with a recurrence will die from it, but let us assume a poor outcome with a 50% mortality from recurrence. This means that two recurrences must be prevented to save one life. It would appear that the cost of saving a life is not £20,000 but nearer £1 million.

New formulations of older drugs are often promoted by manufacturers, sometimes around the time that the patent on the older drug has expired. Bear in mind that some of these new drugs may benefit the manufacturer more than the patient.

"New for old" drugs include:

  • "Me too" drugs which are similar to an existing product by another manufacturer.
  • "Second-generation" products such as escitalopram or desloratadine. These are isomers or metabolites of an existing drug, and may be marketed as "more effective" (because a lower dose is required on a weight for weight basis). In practice there is seldom any clinical benefit.
  • New formulations such as slow-release preparations - these may sound useful, but in practice the advantage may be minimal.

A review of these products in the Drug and Therapeutics Bulletin concluded that "new for old" drugs are seldom worth the extra expense.[21]

Further reading and references

  • Stacey E, Toun Y; Critical reading questions for the MRCGP. BIOS Scientific Publishers, 1997 Useful summary of critical reading skills for doctors, with examples

  • Greenhalgh T; Papers that summarise other papers (systematic reviews and meta-analyses). BMJ. 1997 Sep 13315(7109):672-5.

  1. Greenhalgh T; How to read a paper. Papers that report drug trials BMJ 1997 Aug 23

  2. Gale EA; Lessons from the glitazones: a story of drug development. Lancet. 2001 Jun 9357(9271):1870-5.

  3. Licensing of medicines, Medicines and Healthcare products Regulatory Agency (MHRA)

  4. Freemantle N; Commentary: What can we learn from the continuing regulatory focus on the BMJ. 2010 Sep 6341:c4812. doi: 10.1136/bmj.c4812.

  5. Cohen D; Rosiglitazone: what went wrong? BMJ. 2010 Sep 6341:c4848. doi: 10.1136/bmj.c4848.

  6. British National Formulary

  7. Bisson JI; Post-traumatic stress disorder. BMJ. 2007 Apr 14334(7597):789-793.

  8. Good Medical Practice (2013); General Medical Council

  9. Milne CP, Bruss JB; The economics of pediatric formulation development for off-patent drugs. Clin Ther. 2008 Nov30(11):2133-45.

  10. TOXBASE®

  11. British National Formulary for Children; British Medical Association and Royal Pharmaceutical Society of Great Britain. London

  12. Greenhalgh T; How to read a paper. Getting your bearings (deciding what the paper is about). BMJ. 1997 Jul 26315(7102):243-6.

  13. Greenhalgh T; How to read a paper. Statistics for the non-statistician. I: Different types of data need different statistical tests. BMJ. 1997 Aug 9315(7104):364-6.

  14. Bekelman JE, Li Y, Gross CP; Scope and impact of financial conflicts of interest in biomedical research: a JAMA. 2003 Jan 22-29289(4):454-65.

  15. van der Meer JW, de Gier AM, van Swaaij WP, et al; Independent medical research. Neth J Med. 2007 Apr65(4):124-6.

  16. Becker A et al.; The association between a journal’s source of revenue and the drug recommendations made in the articles it publishes. Published online ahead of print February 28, 2011 CMAJ 10.1503/cmaj.100951

  17. Hargreaves S; Ghost authorship of industry funded drug trials is common, say researchers. BMJ. 2007 Feb 3334(7587):223.

  18. Lenzer J; Truly independent research? BMJ. 2008 Aug 21337:a1332. doi: 10.1136/bmj.a1332.

  19. Akobeng AK; Communicating the benefits and harms of treatments. Arch Dis Child. 2008 Aug93(8):710-3. Epub 2008 May 2.

  20. Moore A; What is an NNT? What is ...? series, Hayward Medical Communications, 2009

  21. No authors listed; New drugs from old. Drug Ther Bull. 2006 Oct44(10):73-7.

I have been very ill for 6 years nausea, vomiting, extreme pain in my abdomine, ribs, back, and hip all on my left side after having 17 scopes done. I have had multiple pain injections and they are...

Health Tools

Feeling unwell?

Assess your symptoms online with our free symptom checker.

Start symptom checker