[p2p-research] Fwd: Open Peer Review

[Kevin Carson]

Thanks, Meir.  I'm forwarding this to the P2P email list, because it
seems like a perfect example of the curating functions of the Web at
its best.  Certainly undermines the whole "Library of Babel" critique
of people like Andrew Keen.  I guess the decisive factor will be when
a majority of academicians start paying more attention to such online
reputational systems than to traditional peer review.

---------- Forwarded message ----------
From: meir israelowitz <misraelowitz at yahoo.com>
Date: Aug 24, 2010 2:07 AM
Subject: Open Peer Review
To: Kevin Carson <free.market.anticapitalist at gmail.com>

http://www.nytimes.com/2010/08/24/arts/24peer.html?_r=1&hp


Scholars Test Web Alternative to Peer Review

 For professors, publishing in elite journals is an unavoidable part
of  university life. The grueling process of subjecting work to the
up-or-down judgment of credentialed scholarly peers has been a
cornerstone of academic culture since at least the mid-20th century.



 Now some humanities scholars have begun to challenge the monopoly
that  peer review has on admission to career-making journals and, as a
 consequence, to the charmed circle of tenured academe. They argue
that  in an era of digital media there is a better way to assess the
quality  of work. Instead of relying on a few experts selected by
leading  publications, they advocate using the Internet to expose
scholarly  thinking to the swift collective judgment of a much broader
interested  audience.

 “What we’re experiencing now is the most important transformation in
our  reading and writing tools since the invention of movable type,”
said  Katherine Rowe, a Renaissance specialist and media historian at
Bryn  Mawr College. “The way scholarly exchange is moving is radical,
and we  need to think about what it means for our fields.”

 That transformation was behind the recent decision by the prestigious
60-year-old Shakespeare Quarterly  to embark on an uncharacteristic
experiment in the forthcoming fall  issue — one that will make it, Ms.
Rowe says, the first traditional  humanities journal to open its
reviewing to the World Wide Web.

 Mixing traditional and new methods, the journal posted online four
essays not yet accepted for publication, and a core group of experts —
 what Ms. Rowe called “our crowd sourcing” — were invited to post
their  signed  comments  on the Web site MediaCommons, a scholarly
digital  network.  Others could add their thoughts as well, after
registering  with their own names.  In the end 41 people made more
than 350 comments,  many of which elicited responses from the authors.
The revised essays   were then reviewed by the quarterly’s editors,
who made the final  decision to include them in the printed journal,
due out Sept. 17.

 The Shakespeare  Quarterly trial, along with a handful of other
trailblazing digital  experiments, goes to the very nature of the
scholarly enterprise.  Traditional peer review has shaped the way new
research has been  screened for quality and then how it is
communicated; it has defined the  border between the public and an
exclusive group of specialized  experts.

 Today a small vanguard of digitally adept scholars is rethinking how
knowledge is understood and judged by inviting online readers to
comment  on books in progress, compiling journals from blog posts and
sometimes  successfully petitioning their universities to grant
promotions and  tenure on the basis of non-peer-reviewed projects.

 The quarterly’s experiment has so far inspired at least one other
journal — Postmedieval — to plan a similar trial for next year.

 Just a few years ago these sorts of developments would have been
unthinkable, said Dan Cohen, director of the Center for History and
New  Media at George Mason University. “Serious scholars are asking
whether  the institutions of the academy — as they have existed for
decades, even  centuries — aren’t becoming obsolete,” he said.

 Ms. Rowe, who served as guest editor for The Shakespeare Quarterly’s
special issue  devoted to Shakespeare and new media, said:  “The
traditional process is not so much a gold standard but an effective
accommodation to the needs of the field.  It represents a settlement
for  a particular moment, not a perfect ideal.”

 Each type of review has benefits and drawbacks.		

 The traditional method, in which independent experts evaluate a
submission, often under a veil of anonymity, can take months, even
years.

 Clubby exclusiveness, sloppy editing and fraud have all marred peer
review on occasion. Anonymity can help prevent personal bias, but it
can  also make reviewers less accountable; exclusiveness can help
ensure  quality control but can also narrow the range of feedback and
participants. Open review more closely resembles Wikipedia  behind the
scenes, where anyone with an interest can post a comment.  This
open-door policy has made Wikipedia, on balance, a crucial  reference
resource.

 Ms. Rowe said the goal is not necessarily to replace peer review but
to use other, more open methods as well.		  In some respects
scientists and economists who have created online repositories for
unpublished working papers,  like repec.org, have more quickly adapted
to digital life. Just this month, mathematicians used blogs and wikis
to evaluate a supposed mathematical proof in the space of a week — the
scholarly equivalent of warp speed

 In the humanities, in which the monograph has been king, there is
more  inertia. “We have never done it that way before,” should be
academia’s  motto, said Kathleen Fitzpatrick, a professor of media
studies at Pomona  College.


 Ms. Fitzpatrick was a founder of the MediaCommons network  in 2007.
She posted chapters of her own book “Planned Obsolescence” on  the
site, and she used the comments readers provided to revise the
manuscript for NYU Press. She also included the project in the package
 she presented to the committee that promoted her to full professor
this  year.

 Many professors, of course, are wary of turning peer review into an
“American Idol”-like  competition. They question whether people would
be as frank in public,  and they worry that comments would be short
and episodic, rather than  comprehensive and conceptual, and that
know-nothings would predominate.

 After all, the development of peer review was an outgrowth of the
professionalization of disciplines from mathematics to history — a way
 of keeping eager but uninformed amateurs out.

 “Knowledge is not democratic,” said Michèle Lamont, a Harvard
sociologist who analyzes peer review in her 2009 book, “How Professors
 Think: Inside the Curious World of Academic Judgment.” Evaluating
originality and intellectual significance, she said, can  be done only
 by those who are expert in a field.

 At the same time she noted that the Web is already having an
incalculable effect on academia, especially among younger professors.
In  her own discipline, for instance, the debates happening on the
site Sociologica.mulino.it “are defined as being frontier knowledge
even though they are not peer reviewed.”

 The most daunting obstacle to opening up the process is that
peer-review  publishing is the path to a job and tenure, and no
would-be professor  wants to be the academic canary in the coal mine.

 The first question that Alan Galey, a junior faculty member at the
University of Toronto, asked when deciding to participate in The
Shakespeare Quarterly’s experiment was whether his essay would
ultimately count toward tenure. “I went straight to the dean with it,”
 Mr. Galey said. (It would.)

 Although initially cautious, Mr. Galey said he is now “entirely won
over  by the open peer review model.” The comments were more extensive
and  more insightful, he said, than he otherwise would have received
on his  essay, which discusses Shakespeare in the context of
information theory.

 Advocates of more open reviewing, like Mr. Cohen at George Mason
argue  that other important scholarly values besides quality control —
for  example, generating discussion, improving works in progress and
sharing  information rapidly — are given short shrift under the
current system.

 “There is an ethical imperative to share information,” said Mr.
Cohen,  who regularly posts his work online, where he said thousands
read it.  Engaging people in different disciplines and from outside
academia has  made his scholarship better, he said.

 To Mr. Cohen, the most pressing intellectual issue in the next decade
is  this tension between the insular, specialized world of expert
scholarship and the open and free-wheeling exchange of information on
the Web. “And academia,” he said, “is caught in the middle.”




-- 
Kevin Carson
Center for a Stateless Society http://c4ss.org
Mutualist Blog:  Free Market Anti-Capitalism
http://mutualist.blogspot.com
The Homebrew Industrial Revolution:  A Low-Overhead Manifesto
http://homebrewindustrialrevolution.wordpress.com
Organization Theory:  A Libertarian Perspective
http://mutualist.blogspot.com/2005/12/studies-in-anarchist-theory-of.html