Comments on the draft Levels of Evidence

i love all of 'em but one, and am not sure i understand that one.

it's the one that describes "positive and negative tests were verified using separate reference standards." hunh? would this apply where negative test results were presumed to be normal and never validated (eg, Goldschlager's gang considered doing angiograms on pts with normal exercise ECGs unethical, and assumed normal exercise ECGs meant normal coronary arteries)?

i guess you are penalizing the SYSTEMATIC non-application of the reference std to all patients (level 3b) by knocking it down an additional peg.

the basis for my disagreement with your proposal to downgrade the use of two different reference standards for the presence and absence of a single target disorder is as before, and dawned on a bunch of us in the hamilton clotting-mafia as we tried to develop better diagnostic tests for clinically-suspected pulmonary embolism:

1. what's the best reference standard for a clinically important pulmonary embolism? i reckon we'd all agree that it's a positive pulmonary angiogram in an appropriately symptomatic patient.

2. but what's the best reference standard for the absence of a clinically important pulmonary embolism? well, given the frequency with which PEs can break up and dissipate by the time we arrange for the angiogram, we decided that a negative pulmonary angiogram would be nowhere near as valid a reference standard as a symptomless 6 month follow-up on NO antithrombotic Rx.

in this case the highest level of diagnostic validity demands different reference standards for patients with and without the target disorder.

Martin Dawes adds

Maybe its time we stopped talking about diagnostic tests and talked about inclusion and exclusion tests (with different gold standards for each - there is a similar problem for establishing negative gold standards for a lot of diseases). At least then people would get used to the idea that they are not necessarily( rarely) the same which for many clinicians is a novel approach.

Chris Ball comments

Klim MacPherson comments:

I found this interesting and very useful.

I would myself think twice before lumping 'Therapy' with 'Prevention' . There are all sorts of different issues in evaluating prevention strategies not least: sometimes their community nature, the real possibility of contamination of the controls, the autonomy of the 'subjects' giving rise to complex issues of preference, possibilities of serious effect modification outside strictly biological mechanisms etc etc . My concern is that Prevention may need variations or changes on the evidence based paradigm which adequately takes account of these issues - otherwise the evidence perforce seems relatively barren.

Bob Phillips' response to this was

We have looked quite closely at the axes (columns) involved, and have come to the conclusion that even if often impractical, prevention studies should be assessed as therapeutic studies.

Klim MacPherson made a futher comment;

I think the issues cannot be dealt with by implying that public health is just another clinical discipline - it is not.

It concerns essentially fit people for whom small (often delayed) benefits have massive effects on the health burden of communities. These people are autonomous (i.e are not patients) and have priorities among which health is unlikely to be foremost.

This does lead to all sorts of different problems - the most acute of which is that randomistion between important preferences is not possible and, if these preferences effect efficacy and outcome, then randomisation (ignoring them) yields attenuated and biased results. But there are other intrinsic problems too.

Benjamin Djulbegovic comments:

Love them or hate them, but the idea that not all evidence is created equal is the lasting legacy of EBM. I have been using your system for some time now, and I think it is the best that is currently available. However, there are some inconsistencies that should be dealt with. The main one is that reference to the design architecture of the studies (which serves as a basis for the ranking system) is mixed with notions such as "outcome research", ecological studies, etc. I suggest that you stick with the design of the studies and rank evidence according to the magnitude of bias that one can expect with a particular study design (e.g. it is clear when you rank evidence as SR or RCT, individual RCT with narrow CI, All or none, individual RCT with wide RCT, individual cohort study etc, but is not clear what is meant by "outcome research", "ecological studies", "first principles, etc). If you have in mind a particular technique (design) commonly used in outcome research, then to use it instead. Otherwise, I suggest to avoid these terms (this would be similar to using term "clinical trials" to replace RCT, cohort studies, etc)

My second suggestion relates to the ranking of "Economic and decision analysis". For example, it is really not clear what constitutes level 1b or 2b evidence (and less clear what is level 1a or 2a- could you provide any example of level 1a according to your definitions?). The problem here is that there are various modeling techniques in decision analysis and which one should be applied depends on the nature of the problem. Perhaps you can use published guidelines on C-E analysis to improve your system (see JAMA 1996;276:1339-41; JAMA 1996;276:1253-8).

Finally, would you dare to recommend how much evidence is needed to make "grade A recommendations". Note, for example, that FDA requires "at least two adequate and well-controlled studies, each convincing on its own" or " a single multicenter study of excellent design" to approve a new drug in the US (obviously this is somewhat different issue, but not in principle...). It would be great if you can illustrate each of your level of evidence with the example (link to a particular reference), so that readers are not in doubt what is meant here.

Andrew Moore commented:

Martin Dawes replied;

and Dave Sacket commented

Re 1: although study size is vital, it's vital to precision, not validity

Mike Clarke commented;