Struggling with the peer review system

[injuryupdate]

D R J V O L U M E 2 3 ? I S S U E 2 ? A U T U M N 2 0 0 5 Sport Health

http://www.injuryupdate.com.au/image...peerreview.pdf

One of the modern pet hates of
clinician researchers (particularly
those who have had this title thrust
upon them by the demands of the
ACSP training program rather than
their own choosing) is the peer-
review process of scientific journals.
A study which might literally take
months to years to perform, and fully
weeks to months to write up, can
be dismissed by a journal editor and
reviewers seemingly within minutes
(once they have finally got around
to reviewing your paper). In recent
years, the bar seems to be set higher
and higher, with the only redeeming
feature being that the journal process
for rejection is quicker than it once
was, thanks to the elimination of
sending manuscripts through the mail.

The answer to the question ?Why do
journal editors reject so many papers??
can be phrased simply as ?Because
they can?, or can be more graphically
compared to a similar question whose
answer seems obvious, such as, ?Why
do good-looking thin young girls
reject so many guys who ask them out
on dates??

What really annoys potential authors
to distraction though, is that the
goal posts don?t seem to be in the
same place for each paper that
gets reviewed. When you read the
reviews of your rejected paper, it
seems fair enough when comments
get made like ?This wasn?t truly a
randomised double-blind placebo-
controlled study with large numbers
and long follow-up, which would
have been preferable for assessing
this form of management?. This is
until you open up the next edition
of the journal that rejected you and
see that the opening article is from
a surgeon from Buttcrack University
Medical Centre who reviewed 18
patients after surgery (from the ones
who bothered to present back to his
office) and found that they all had an
?excellent? result. Somewhere in the
conclusion you could swear that you
read a sentence which said ?The most
pleasing aspect of this study is that
insurance companies will continue to
pay huge rebates for me to perform
this operation in the future, ensuring
that I will be able to buy a lodge at
the Buttcrack ski resort to go with the
beach house I already own?.

Yes, the peer-review process is most
definitely a lottery, and fortunately
I am in the position where I can
afford to laugh about it. It upsets
me that for ACSP registrars, getting a
rejection from a prestigious journal
has an equal effect on the candidate
as failing an exam. If the peer-review
process was a lot more objective, it
might be a fair method of assessment,
but it?s even more subjective than,
say, a viva exam! I contend that there
are many excellent papers that have
struggled to make it into peer review
and some shockers that you can
find using Medline. Just about my
favourite of all time (with some very
famous names amongst the authors) is
?The meniscus as a cruciate ligament
substitute. Collins HR, Hughston JC,
Dehaven KE, Bergfeld JA, Evarts CM.
J Sports Med. 1974 Jan-Feb;2(1):11-
21? which describes ?successfully?
ripping ACL-deficient patients? menisci
out and using them to attempt to
reconstruct the ACL.

Just as Michael Jordan once said
?you miss 100% of the shots that you
never take?, the only thing worse than
not having a paper published in a
PubMed-listed journal is not writing
it in the first place. I am often asked
how I manage to get so many papers
published while still having an active
clinical practice and looking after a
professional football team. One of the
key answers is that I don?t waste time
in the peer-review process. I?m quite
happy to submit a paper to a journal
and, with very few exceptions, if it
doesn?t get accepted first go (with
minor revisions) then I shoot it off
to Sport Health, the New Zealand
Journal of Sports Medicine, Sportslink
or even just put it on my website at
www.injuryupdate.com.au.

The beauty of the Internet is that
the cost of on-line publication has
become very cheap (even if print
publication is more expensive than
ever). I think eventually that even
scientists will cotton on to the fact that
a Google search is just as important
(not relevant but important) as a
PubMed search on a topic of scientific
interest. The most obvious example of
this is when you have a topic where
a medical decision must be made.
Too often all the scientific literature
tells you is that nothing is proven
beyond reasonable doubt, yet in the
real world you still need to make a
decision. With a lot of sifting through
chaff, there are still some pearls out
there on the internet that you can?t
find in a PubMed search.

Fortunately I write enough stuff that
does get accepted in PubMed-listed
journals so that my legitimacy as a
researcher is not in question. The way
the major journals are heading makes
me think I will have less and less
time in the future to be able to devote
to playing their (often unnecessary)
games. Probably heading this list
is ethics committees. I have been
through an ethics committee a handful
of times and I would just about
rather spend my next holiday in India
drinking putrid river water. I agree that ethics committees need
to exist, because some researchers
have to kill animals or make humans
take Vioxx in order to get their study
completed and there needs to be
a check on whether these studies
are justified. However, a good
proportion of studies are intrinsically
ethical and don?t need a politically-
correct committee to pick them to
pieces. They particularly don?t need
25 extra pages to be filled in and
for informed consent forms to be
translated into tribal Aboriginal before
a study will get passed. I don?t think
that a collective review of patient
data (without revealing the names
of patients) should require a signed
consent before it can be done, for
example, but an ethics committee
is sure to disagree. If an ethics
committee was asked how many
people lived in Australia, they would
surely take 12 months to come up
with the answer of ?6 million people,
not counting those who chose not to
participate in the study?.

The rest of this article will describe a
paper which follows using aprotinin
injections in tendinopathy. Put me in
jail for longer than Rodney Adler if
you like, but I didn?t have this study
approved by an ethics committee.
It may amaze those of you in ivory
towers, but those patients who
participated were happy to be asked
for follow-up information without
needing to sign a consent form. I
hope that many readers find this
paper interesting. Some of you might
? others might find more interesting
the reviewer comments, posted
below, from a PubMed-listed journal
which rejected this paper.

I wasn?t surprised with the comments.
They are all pretty consistent. I think
it is likely that if I submitted this paper
elsewhere that it may also be rejected
with similar comments. I wanted to
write this paper up because I noticed
that a significant number of my
patients I was treating with aprotinin
were getting allergic reactions. This is
pretty important information.

Some of the reviewers have said that
I should have just written up these
cases and basically made the paper ?a
case series of allergic reactions caused
by aprotinin injections?. However, the
whole point of giving an aprotinin
injection is not to cause an allergic
reaction but to try to improve results
in tendinopathy. I fully appreciate
that a case series, even with largish
numbers, showing that patients are
pretty happy with their injection
outcomes is only level 3-4 evidence of
efficacy. However, in a world where
level 1 and 2 evidence is so rare, it
reassures me that patients are mainly
satisfied with the treatment I am
giving them after follow-up.

Yes, read this paper with a grain of
salt. The good clinical results may
be due to placebo effect, the natural
history of the condition or simple
prolotherapy which could have
been achieved with an injection of
glucose instead. At least there is some
encouragement to do more RCTs.

The funny thing is that patients don?t
really give a rat?s proverbial about
RCTs. At the time I was treating
dozens of patients for Achilles
tendinopathy with aprotinin in a
non-randomised fashion, we (Richie
Brown and I) were trying to cast a
wide net to recruit patients into an
RCT. The patients all wanted to know
?How are the other patients doing
who have had this treatment?? and,
when the answer was ?Generally
well?, they mainly wanted to just get
the aprotinin injection rather than be
enrolled in a study where they have
a 50% chance of getting the injection.
Being someone who does follow
the tendinopathy literature fairly
closely, of course I offered them other
treatments as well, many of which
they had previously tried and had
failed.

In terms of hard-core science, the
take-home message of the following
paper is that aprotinin has the
potential to cause allergic reactions
even with local injections. However,
it seems ridiculous to throw out the
results of subjective follow-up of
patients just because the methodology
is not as good as it could have been.
Therefore I would much rather
publish this paper as is in Sport
Health than put a chainsaw through
it and send it back in to the big time
journals.

It is also worthy of note that some
excellent quality papers describing
efficacy of aprotinin injections in
tendinopathy have been published by
the Capasso group, but these don?t get
considered in Cochrane reviews etc,
because they have appeared in non
PubMed-listed journals.

Anyway, next time you collect an
interesting clinical case series of three
patients with a rare condition, or
collate two year?s worth of pre-season
fitness parameters from a sports
team, why not send it in to Sport
Health or give it to me to post (with
acknowledgement) on injuryupdate.
com.au? I agree it is research of a
quality that will probably bounce from
the majors, but so-called low level
research is better than none at all.
After all, it?s what we base at least half
of our clinical practice on!

Reviewer comments regarding the
aprotinin paper from peer-review:
Reviewer 1 Comments
This work needs so much correction/revision
and conceptual change that I think it should
be rejected. The paper purports to study
the risks of aprotinin injections for chronic
tendinopathy. It begins to but wanders off
into discussing its efficacy. We don?t use this
drug in this country for tendinopathy (usually
it?s used for blood conservation/hemostasis in
cardiac or joint replacement or spine surgery).
The part on efficacy is fraught with numerous
inconsistencies with numbers, no consistent
method of usage, the results are jumbled and
almost anecdotal in reporting. The risk part
starts with known data (RE: Beierlein, W. Ann
Thorac Surg 2005) and tries to compare with
that. The level of evidence here is very low.
The technique of application is not discussed
(peri-tendinous or intra-tendinous). The risks
or side affects are poorly described (7 cases
of systemic allergic reaction occurred but not
described as to specifics). Adequate follow-
up is not defined. Multiple conditions were
treated in the same patient and the numbers
in results were not consistent. There was a
20% no response. In short, I think too many
obstacles to overcome.

1) make this a study of risks only
2) the numbers reported must ?add up? to be
consistent
3) telephone contacts are not adequate.
4) need defnitions eg. protocol, systemic
allergy, what is ?substantially improved??
5) what was technique of administration?
6) a control (another med or saline) would be
helpful for comparison. e.g. Capasso et al.,
Sports, Exercise, and Injury article.

Reviewer 2 Comments
The authors report on a diverse group of
patients who undergo aprotinin injection for
treatment of chronic tendinopathy. Although
this is a fairly large number of consecutive
patients, and the response rate is reasonable
at 80%, there are some major flaws in
this study. The patient population is quite
diverse including a special subpopulation of
professional athletes. Additionally the injury
being studied is not well defined. The authors
used the term ?tendinopathy? and do not
distinguish between tendonitis, tendinosis and
peritendinitis. They do not discuss how the
diagnosis is formalized or whether radiologic
modalities such as ultrasound or MRI utilized.
There are also multiple different injury sites
involved. There is no clear discussion of the
indications for a first injection or subsequent
injections. And finally the outcome criteria are
very weak. As this is an uncontrolled study, I
think using a purely subjective patient derived
questionnaire as a sole form of outcome data
is quite weak. While the information on the
incidents of allergic reaction does appear to be
valuable information, it has already reported in
the literature. Based on this weak study design
and the fact that this is not new information, I
do not think that this manuscript is suitable for
publication in this journal.
Specific Comments for the Authors:
The authors state that this is a case review
and follow up of a 121 consecutive patients.
I think it would be more accurate to say that
this is a retrospective case review and mention
that follow up was obtained through mailed
questionnaires.

The authors should mention whether the
institutional review board at their hospital
approved this study.

It would be helpful if the authors would clarify
the inclusion criteria for their study. They
say that there are 155 different tendinopathy
cases. Does this mean tendinitis? Tendinosis?
Peritendinitis? How was the diagnosis made? Is
this purely a clinical diagnosis or were MRIs or
ultrasounds performed in some cases?

I?m curious whether the authors noticed any
correlation between the anatomic location of
injection and results.

What were the indications for aprotinin
injection? Was this offered to all patients with
tendinopathy, or only if they failed to improve
with rehabilitation? Also what were the
indications for a second injection?

The authors state that 8 cases were followed
up for less than 3 months. Clearly this is
not an acceptable length of follow up to
determine ?success?.

The authors state that ?in general, there was
good progress of the patients? conditions?.
I don?t think it?s legitimate to make this
statement based strictly on patients? objective
assessments.

Reviewer 3 Comments
According to the title of this paper, its main
purpose is to report the risks of aprotinin
injections for chronic tendinopathy. The
specific risk that the authors cite is the risk
of allergic reaction. However, much of
the manuscript is devoted to a moderately
expansive description of the patient
population and the efficacy of the treatment.
The work was not really designed to report on
efficacy and therefore this aspect of the paper
is very problematic.

It appears that the authors did not really plan
to be doing a study at all, but were using
aprotinin extensively and noticed allergic
reactions, so decided to report them. As a
treatment study, the paper is very spotty in
the available information, and does not have
the strength of methodology to comment on
the efficacy of the treatment. Follow-up is
subjective, it doe not seem that concomitant
treatments such as PT were controlled, and
there is no control group. In order to report
on the possible efficacy of this off-label use of
a drug, the authors should perform a RCT.

The allergic reactions could be separated out
and submitted as a series of case reports, with
the number of cases treated during the time
period being delineated so that the incidence
of these complications could be estimated.
This might serve as a warning to those who
might wish to use this treatment on their own.?

[sweveSek73]

im tempted to get involved with this but considering the tremendous variance in prison conditions in this country, it seems fruitless.

ntw...iirc, sheriff joe feeds his inmates for a dollar a day.