Inspired by Erick Turner I wrote these two rapid reviews based on FDA reports:
I produced these reviews, not to claim perfection but to familiarise myself with the type of content and to try to see what a report might look like. After I’d finished them I sent the links to Erick who, very generously, supplied some really useful feedback which I reproduce below (with his permission). I consider myself incredibly lucky to have had Erick’s considerate teaching on this matter!
Background: whether in the product labeling or journal articles, the mechanisms always sound very “scientific”, and my hunch is clinicians are overly impressed by that, to the point that they perceive the drug to be effective before they’ve even been presented with any clinical trial evidence. Additionally, oftentimes some things are mentioned that may be of no relevance to whether the drug works. Finally, despite the fact that the words may be foreign to the average clinician, the drug in question may be a me-too drug.
The two graphs you have there have no error bars, and there are no P values so one doesn’t know whether the differences are statistically significant.
BTW, even negative studies have somewhat similar-looking graphs. There’s always an impressive reduction from baseline, but for placebo as well as active drug.
What such change-from-baseline graphs don’t show is how far the “average” patient might be from a score that might be meaningful. I have on occasion, redone graphs so that the y-axis shows absolute score.
I’m not advocating for this, as this would involve extra work, and my complaint here hasn’t been talked about in the peer-reviewed literature (yet), as far as I know. Just mentioning it as a further limitation of such graphs.
By contrast, the graph that we looked at together as we talked focused on the end-of-study values and showed the 95% confidence intervals around the point estimates. So that allowed you to see whether the results were statistically significant and how they compared for the different groups.
More importantly, they provided a sense of CLINICAL significance. As I’m sure you well know, there can be statistical significance without clinical significance.
This becomes an issue when you consider the the FDA’s threshold for approval rests primarily on statistical, not clinical, significance is 2 or more studies — but the magnitude of the effect may be quite small.
So I would recommend either:
1.directly presenting something that shows both types of significance. That could be the (a) raw unstandardardized change from baseline, like that graph we looked at together. Or (b) it could be a standardized effect size, of the type I showed in my NEJM article and others have used, either a Hedges’s g or Cohen’s d.
2.Showing enough statistics in a summary table that would allow researcher(s) doing meta-analyses to undertake (1). An example would be Table 3 on page 13 of 99 of the statistical review (the page preceding the graphs you showed). The stats have been streamlined (or stripped down) into Table 9 on page 21. What’s been omitted are the sample sizes and baseline scores, which could frustrate some researchers (and possibly some clinicians, who often like to know whether the study was big or small. Yet another table to consider would be Table 19 on page 54/99 of the stat review. However, it includes a “key secondary efficacy endpoint” — but I would try to suggest adhering to a primary-only policy.
Interesting point about the mortality for the drug for DM.
One possibly related comment, probably obvious, is that when we talk about “primary outcome”, we’re usually focused only on the efficacy side of the risk-benefit ratio. The FDA looks separately as the two issues of efficacy and safety, and a problem with either one of them can derail approval. (In the DM drug case, it appears that the distinction is more blurry, in contrast to within the world of psychotropic drugs.)
A lot of drug SRs and MAs focus only on efficacy — that’s all I’ve done in my papers, for example.
But when a clinician is deciding whether to add a drug to his/her arsenal, he/she needs to know about safety, as well.
All the safety data you want should also be within the FDA review—you’re certainly not going to find more safety issues in the published literature. It’s just a matter of whether you want to go after that, as well, so just have to decide on your scope.
In asking his permission to reproduce his reply he suggested I include the following table to better illustrate the data and show that you can utilise the data for meta-analysis purposes.