Clinical Trials under the microscope. Before you part with your hard earned cash you should be able to tell whether the headline grabbing statistics are worth the paper they are written on.
A clinical trial suggests accurate, trustworthy results brought about through exhaustive expert analysis. But when it comes to something as important as the effectiveness and safety of supplements, can we afford to simply assume all clinical research is of equal merit?
A study published in the Journal of the American Medical Association (JAMA) in 2012 found that the number of registered clinical trials had sky rocketed in recent years… but that many of the practices and motives in some of this work was highly questionable.
By reviewing as many ingredients and products as we have, we’ve had to read a lot of the research. We decided to write this article to better explain how we separate the good research from the bad.
A clinical trial is a test; the first practical test on human subjects. Up to that point the research would have been theoretical, or conducted on animals.
It normally takes the form of controlled experiments to gauge not only the effectiveness of an ingredient, but also to flag up any potentially harmful side effects.
The first clinical trial is commonly thought to have occurred around 1747 when, to the delight of sea faring stereotypes everywhere, James Lind, a Royal Navy doctor, employed a method not unlike that still used today to successfully identify vitamin C as an effective treatment for scurvy.
He divided a group of afflicted sailors into pairs and gave a different remedy to each, observing that only those given oranges and lemons made a dramatic recovery.
The fact that he had also given some participants sulphuric acid and killed them arguably made the technique’s first outing a bit of an ethical score draw overall, but with a few murderous wrinkles ironed out this would continue to serve as the blueprint for most evidence-based research.
Despite science having now had 268 years to get its act together and come up with something better, modern day clinical trials are still our most efficient means of determining exactly what works and what doesn’t when it comes to our health and well being.
The word clinical rather suggests long established, universally accepted and standardised procedures being used to obtain results.
Whilst in most quality clinical trials this will prove to be the case, the 2012 JAMA study (‘Characteristics of Clinical Trials Registered in ClinicalTrials.gov, 2007-2010’), found that across a data set of 96,346 clinical studies, there were significant inconsistencies in methodology.
That’s why in assessing the scientific basis for a supplement or ingredient’s claims, we never merely take it for granted that the fundamentals were in place.
First check whether a control group was used in the course of study. Say, for example, the manufacturer of a new supplement wants to demonstrate their product is able to boost testosterone and improve strength. That company might take a sample of weightlifters and give them all regular doses of the product.
If over a period of time improvements are recorded, would that seem a reasonable basis for their claims. No, is the answer, this would constitute incredibly weak evidence. It barely even qualifies as a clinical trial.
After all, who’s to say if the element being tested was genuinely the catalyst for improvement? When people lift weights regularly, they tend to improve naturally; perhaps it was merely normal progress which was being wrongly attributed to the supplement.
To negate any doubt about the outcome, researchers in a good clinical trial will typically select a cross section of closely matched weightlifters and split them into two groups.
Only one group is given the active ingredient, while the other, the control group, is given a placebo which alters nothing. The control group now provide baseline stats against which improvements in those taking the new ingredient can be accurately judged.
Next look at whether or not the trial was randomised. A higher quality clinical trial will try to use volunteers who are as closely matched as possible to cut down on the extent to which naturally occurring individual differences may influence results.
However unless you are lucky enough to find the world’s first family of hectuplets (set of 100 twins – if it’s not a word it should be) and by happy coincidence they happen to fall within a demographic relating to your study, chances are there will always be slight variations within any sample.
When a trial is described as randomised it means a computer programme, not the researcher, has assigned how contributors are grouped, helping to cut down on any bias from those conducting the study.
For instance, let’s say a study is being conducted using men with low testosterone; though they will all have lower than the average levels, some may still have slightly higher amounts than others taking part.
In a trial which is not randomised, whoever is running the study, keen for their testosterone enhancing agent to succeed, may unconsciously put the men with marginally lower testosterone in the control group, hence giving those taking the active treatment an edge before the trial has even begun.
I generously said ‘unconsciously’ because, well, that’s just the kind of guy I am, but of course we know there’s always the chance that in more unscrupulous trials these subtle advantages are given deliberately by those with a vested interest in testing going well. More of which later.
It’s common, or at least it should be, for clinical trials to be ‘blind’ i.e. participants left unaware whether they are taking the placebo or ingredient for the duration of research. Ideally though, a double blind trial is desirable.
A double blind trial is where the researchers themselves are similarly in the dark as to which participants are receiving active treatment, and they remain that way until the investigation is concluded. This is achieved in much the same way, and done for many of the same reasons, as randomisation.
As Gene Wilder and Richard Pryor proved in their 1989 movie ‘See No Evil, Hear No Evil’, when neither party is in full possession of all the information at any one time the results can be farcical and often hilarious.
So in order to ensure the validity of double blind study results it is advisable to set up a Data Monitoring Committee (DMC.) A well run DMC (sorry, couldn’t resist) should be comprised of about 3 to 7 professionals, who are completely independent of the trial.
Usually at least one of these members will be a statistician, while the rest might be authorities in the field of study.
They will have access to the all the data and should meet at regular intervals to access a trial’s progress or review ongoing results. Importantly a genuinely effective DMC should have ultimate power to terminate a study where deemed appropriate, usually due to questions over the safety, outstanding benefit and overall futility.
That is to say, is it safe, is it worth it, is it working?
In that same JAMA report, it was discovered that of 47% of the clinical trials investigated were industry sponsored rather than academically motivated. To an extent this is understandable. Certainly the supplement industry can’t always be expected to just sit on its hands until some professor at a university or college decides he or she fancies checking what exactly boosts testosterone.
Provided they are willing to accept and disclose the outcome of research, even in the event of it being unsuccessful, there is no reason why industry commissioned clinical trials can’t be every bit as valuable as their academic counterparts.
That is an ideal world though. In the real world there have been several high profile examples of industry sponsored work with major question marks hanging over them.
If you are trying to evaluate a supplement or ingredient then one more thing you want to pay attention to is this – how is the success measured? I have seen so many studies where the researchers used something which I consider to be subjective.
How many bench presses the subjects could do. How far they could swim. How many squats they could do.
There are a million reasons why you could end up doing more squats. Did the children keep you awake all last night? Did you have your porridge in the morning? How much water did you drink before you went into the gym?. Stressed at work and unable to concentrate? Etc etc, you get the point.
If the idea is to measure testosterone then why not….. measure testosterone? Draw blood and run bloodwork. It seems the obvious thing to do but seemingly it’s not that common. It tends to (understandably) be done when the trial is run in a University Hospital, or when the patients are undergoing the trial as a part of some other study.
It would be much more useful if this were always done.
All trials are not born equal. Be wary of extravagant claims like ‘Proven to increase testosterone by 24% in only 3 days!” and the like. You have a choice, you either dig deeper and spend a lot of time poring through abstracts and the detail (where the devil lurks) of these studies.
Or… you let us do it for you. You’ll find that we quote and summarise the studies in our supplement and ingredient reviews so you can check the facts if you want to.
One comment
Future Alternatives to TRT - Enclomiphene Citrate says:
[…] Let’s Get Clinical: What We Look For In A Clinical Trial […]