Good Study, Bad Study

Good Study, Bad Study

Take us through those numbers Victor.

tl; dr ... Short and to the Point
Who is this article aimed at?
Anybody interested in an comparison of the good and the bad in clinical testosterone studies.
I am busy - summarize it for me

Keep an open mind when supplement manufacturers cite 'clinical research'. Not all studies are born equal.

Examples of Good and Bad Clinical Testosterone Studies

We’ve done a couple of articles on clinical trials in the past. We did this one [what makes a good trial], and more recently a piece on [this meta study]. A meta study, in case you are not aware, is a study on other studies.

In many of our reviews we mention and critique studies – it’s important because the scores we give formulas are based on the ingredients and dosages. And the clinical studies are the way to work out whether the supplement manafucturers have got that right.

So I though we should do a post showing som practical examples of good and bad studies. To help you help yourself when it comes to appraising products and ingredients.

Let’s look at a practical example or two to demonstrate classic examples of research which is not textbook.


ZMA – A textbook ropey clinical study

ZMA is a classic example of a bad clinical study. It wasn’t bad for the company that engineered it of course, they made millions.

ZMA is the three letter acronym for Zinc Monomethionine Aspartate and Magnesium Aspartate. It’s turned into a real Hoover/Sellotape/Xerox style term these days, brand turned to popular term. It was originally a supplement produced by a company named SNAC Systems.

It claims to help you both recover quicker after exertion through deeper REM sleep, and also enhance strength and stamina by boosting testosterone levels. Now it does contain good tst boosting minerals (Zinc and Magnesium) but the issue at stake here is the validity of the clinical trial surrounding it.

It is a patented proprietary blend designed by Victor Conte, founder of Bay Area Laboratory Co-Operative (BALCO.) For a while SNAC’s product sold well, it was even used by some elite athletes as a physical recovery aid; popularity which was doubtless helped by some solid looking clinical research.

The trial most often offered up as proof of its effectiveness is a 1998 study conducted at Western Washington University by Lorrie Brilla Ph.D. Brilla took 27 competitive National Collegiate Athletic Association football players and while 15 were assigned a placebo, 12 were given a nightly dose of ZMA over the course of an intensive 8 week training programme.

The results were overwhelmingly positive. Astonishingly so.

Those taking ZMA experienced an over 30% increases in free and total testosterone levels compared to an over 10% drop in the placebo group. As well as measuring testosterone levels, Brilla also tested the pre and post leg strength and power capacity of the two groups and there was good news for ZMA; the muscle strength of the ZMA group increased by 11.6% compared to a 4.6% increase in those on the placebo.

Likewise, the ZMA group saw twice the functional increase power compared to the control group. A slam dunk.

The Slam Debunk

Pretty conclusive stuff, eh? Independent, academic researchers, recognised methods and clear cut results. Evidence enough for some to heartily recommend we all run out, get ourselves some ZMA and just start chucking handfuls of it back. Thousands did.

In their book ‘Game of Shadows’ Mark Fainaru-Wada and Lance Williams, investigative reporters for San Francisco Chronicle, tell of how, at the same time as Brilla was presenting her findings to the Southwest Chapter of the American College of Sports Medicine, SNAC Systems put out a press release hailing the results.

The release contained a quote from Victor Conte, cited only as ‘co-investigator’ proclaiming that,

“ZMA is the only all-natural product that has been clinically proven to increase anabolic hormone levels and strength in trained athletes.”

What the press release neglected to mention was that Victor Conte in fact owned SNAC Systems, it was merely an affiliate of his BALCO group.

Conte also allegedly helped fund the study and Brilla was assisted in her research by Jim Valente, BALCO vice president and Dr Brian Goldman, the company’s medical director.

The study also failed to disclose that originally 57 football players signed up to take part in the trial and over the 8 week period, tellingly, 30 of them quit.

One of the sad things about all this is that they were simply over-egging the pudding. The longevity of the product (millions of units are still sold every year) means that it must be effective.

Yes, a subsequent trial by a German organisation had inconclusive results but, I’m sorry. Despite what the supplement industry’s detractors might say, you cannot have a completely ineffective product in widespread use for decades. Marketing is a powerful force alright, but it’s not that bloody powerful.

The fact that such dramatic results had to be cooked up by Conte, Brilla and co. just provides ammuniton to those that seek to denounce the supplements industry.

A Good Clinical Study

After that onslaught you might think we view all clinical research with narrowed eyes of intense suspicion. Well we don’t, because;

  1. We know there’s still a huge amount of incredibly worthwhile and massively beneficial research going on out there.
  2. It’s actually quite difficult to read these things while you’re squinting at it through the narrowed eyes of intense suspicion.

There are loads of good examples of course, we picked the following one which has loads of aspects you’d look for in a properly conducted trial. It also demonstrated impressive results.


Mucuna pruriens study – Lucknow, India, 2009

Alternatively known as Lacuna bean, Velvet Bean, or Florida Velvet Bean, Mucuna pruriens is a is a climbing shrub native to areas such as Africa and South East Asia where extracts of it have long been used in traditional medicines.

One of the most notable human studies on Mucuna pruriens and testosterone was published in 2009, when a research team from the Department of Biochemistry at C.S.M. Medical University in Lucknow, India released the results of their clinical trial.

The aim of the study, which was carried out over a two year period between 2005 and 2007, was to better comprehend the workings of a Mucuna pruriens supplementation on male infertility.

In all, 150 participants took part, with 75 in the normal control group and 75 men undergoing infertility screening.

The test group took 5mg of Mucuna pruriens orally every day in a single dose (along with milk) for three months. In all, 150 participants took part, with 75 in the normal control group and 75 men undergoing infertility screening.

The researchers monitored changes in various aspects of their volunteers’ hormonal makeup during the trial. They measured levels of;

  • Serum testosterone, which is pretty self-explanatory.
  • Luteinizing hormone, which stimulates Leydig cells in the testes to produce testosterone, and dopamine, a neurotransmitter associated with the pleasure centres of the brain and responsible for sexual arousal.
  • Prolactin, a protein the body releases to counter the dopamine immediately after sexual gratification to effectively putting the brakes temporarily on sexual arousal.
  • Adrenaline and noradrenaline, bodily responses to stress which can adversely affect production of testosterone.
  • Follicle-stimulating hormone which is a hormone secreted by the anterior pituitary gland which promotes the formation of sperm.

At the conclusion of the trial it was found Mucuna pruriens significantly altered levels in the test group and results strongly suggestive that the supplement could be beneficial to fertility.

Serum testosterone increased in the men with normal sperm counts by 27%. Men with low sperm counts had an average increase in testosterone levels of 38%, whilst men with low sperm motility (also showed an average increase of 38%.

Luteinizing Hormone increased in the normal men by 23% and Prolactin dropped by 19%. In the men with low sperm count Luteinizing Hormone increased by 41% and Prolactin dropped by 32%. Men with low sperm motility saw their LH levels rise by an average of 40% and Prolactin drop by 11%

What Sets This Apart?

So we can say there is sound clinical evidence for Mucuna pruriens being effective in boosting testosterone and aiding fertility. What sets this research apart, and why do we like it?

Firstly there is the ingredient itself, a pure naturally occurring shrub with a strong anecdotal track record of safe health benefits. There can be no real competing corporate interest in ensuring the results are positive, it’s a widely grown shrub.

Mainly, though it’s the method;

  • Sound clinical practices were employed by a team of highly academic specialists. In the intervening years, no one has felt the need to question their results.
  • The study took in a comparatively large sample compared to aforementioned trials which is always a bonus as it lends weight to results.
  • The study took place over a prolonged period. A team that takes two years to compile thorough results can hardly be accused of having ‘a time is money attitude’ or rushing research for commercial reasons.

That’s not to say that all studies have to stretch into years to be well intentioned, of course they don’t, but if they do it can definitely be considered a plus point.

What Does All This Mean For Us?

All in all it’s a bit of a clinical jungle out there now. Amidst soaring numbers of new registered trials, wildly variable in their quality, it’s no longer feasible to simply assume that ‘clinically tested’ automatically equals safe and effective.

That’s why we feel it’s so important, before we even think of making any recommendations, to first spend the maximum amount of time and effort sorting the solid from the suspect as far as the research goes.

It’s all about the clinical research.

We’re only interested in the supplements that have legitimately put in the scientific hard yards before making it to market. If there’s any one word which encompasses everything we look for when faced with a supplement it would probably be, clarity.

Clarity of ingredients of course – so avoiding supplements hiding behind the convenient smokescreen of proprietary blends, papering over the cracks with flash brand names – but also clarity of research. What studies are there to support a product? Who was involved? What were their methods? What were their motives?

After all, the supplement companies who genuinely value your top physical performance as much as their top financial one, are the ones with nothing to hide.

tl; dr ... Short and to the Point
Who is this article aimed at?
Anybody interested in an comparison of the good and the bad in clinical testosterone studies.
I am busy - summarize it for me

Keep an open mind when supplement manufacturers cite 'clinical research'. Not all studies are born equal.