The Web Instead of Traditional Peer Review?

That’s a more accurate title, but I really wanted to call this post, Tenure Online?

First off, I wanted to ask the question, what do professors out there think? Can peer-review be open sourced? Is online work getting any credit, or is it still all about traditional peer reviewed articles?

The prompt for this is an article in the NY Times: Scholars Test Web Alternative to Peer Review

The grueling process of subjecting work to the up-or-down judgment of credentialed scholarly peers has been a cornerstone of academic culture since at least the mid-20th century.

Now some humanities scholars have begun to challenge the monopoly that peer review has on admission to career-making journals and, as a consequence, to the charmed circle of tenured academe. They argue that in an era of digital media there is a better way to assess the quality of work. Instead of relying on a few experts selected by leading publications, they advocate using the Internet to expose scholarly thinking to the swift collective judgment of a much broader interested audience.

The Shakespeare Quarterly is leading the charge over in the Humanities. They handled the open comment process through Media Commons Press, which has the tagline: “Open Scholarship in Open Formats.”

The larger point comes later in the article, and it’s one I hope to hear people’s opinions about:

Today a small vanguard of digitally adept scholars is rethinking how knowledge is understood and judged by inviting online readers to comment on books in progress, compiling journals from blog posts and sometimes successfully petitioning their universities to grant promotions and tenure on the basis of non-peer-reviewed projects.

3 thoughts on “The Web Instead of Traditional Peer Review?

  1. For most of my recent papers, I’ve had far more thoughtful and helpful reviews on blogs and internet forums than from the journals themselves. So I think there is a lot of potential.

    I think that journal articles are in no danger. Review is about authority — most online sources have little, and the few that have authority can’t possibly handle the workload of reviewing.

    I see very little point in edited volumes now, however — why would anybody put their work into a volume that costs $120 and will be read by only twenty people? It will be interesting to see what e-books do to monographs, also — open reviews and cheap (or free) distribution could allow a real renaissance for the short monograph.

  2. Daniel and John —

    Glad that you guys have asked this. I’m personally really torn about new publishing models, my loyalties and interests at right angles to each other. The discussion of academic publishing over at Savage Minds and elsewhere online, for example, focusing on the open access movement and what to do about AnthroSource, the American Anthropological Association’s primary journal database, left me deeply troubled.

    On the one hand, like many in our field, I crave the disciplinary acceptance and recognition that comes from publishing in our leading journals, not merely because I’m burdened with a fragile ego, but also because I care about certain streams of thought, topics, and approaches and want to promote them, see them discussed prominently. I care about anthropology so it matters to me how our field is thinking about crucial problems.

    John’s right, of course, that the authority of academic journals is not likely to decrease anytime soon. If anything, recent moves to try to quantify and rate research output here in Australia have placed even more emphasis on publication in journals with high citation rates or other forms of certification. In Oz, we have a list of journals rated from A* to C, making it abundantly clear where we should publish our research (I posted the list at Culture Matters).

    However, in cultural anthropology, some of the journals still considered so highly in measures of esteem and impact have a number of serious problems: 1) much lower citation rate than journals in other fields, 2) low circulation, 3) pay walls and restricted access, 4) incredibly slow turnaround times, and 5) decreasing numbers of library subscriptions. With all of the effort, creativity and extraordinary ideas that go into these publications (yes, I’m serious), it’s a real shame that they are so obscure most of the time and with for-profit companies publishing more and more of the journals, it’s hard to justify contributing free labour to someone else’s product, especially when you end up having to buy it back.

    I’d love to publish my work in open access publications, but the OA journals in our field are quite small and sometimes precarious, run more often by industrious graduate students than by scholars central to our field (I mean anthropology here, although biological anth is in better shape than the sociocultural folks in some of these issues). We haven’t successfully placed anthropology in the big open access journals (again, with bio anth the exception), so our field shrinks in relative visibility as others become more accessible. It’s hard to see how we can stand still in a changing environment and not lose ground.

    When I recently started looking at the available platforms for publishing an open access anthropology journal, I realized that the one greatest obstacle holding me back (aside from fear of abject failure) was the issue of marshaling colleagues for peer review. I’ll be watching with great interest some of the experiments being done with online, open comment review — I just don’t know how a field can expect to exert any intellectual influence outside its own borders if it hides most of its thinking in hard-to-reach publications and open-access, open-review would be great visibility.

    The irony is that open reviews would actually be a real improvement on the current, private review process. In peer reviewed journals, the peer review process is invisible to the reader in the final product; we can’t see the debates, only the final version of the publication.

    The sad fact is that private peer review can often force the removal of some of the most contentious and interesting points that an author wants to make. As I’ve written elsewhere, when I’ve tried to publish neuroanthropology stuff, if I even get a foot in the door, reviewers usually want me to strip out the neuro-. If the reviewers find an argument to be a sticking point, it has to go, even though it might be very interesting or well worth considering. Especially for junior scholars, needing publications for tenure and promotion, discretion can be the better part of intellectual debate; sticking points are just dropped in the face of opposition.

    Right now, we get a kind of open review, but only piecemeal, in places like John’s weblog (which I use frequently for this purpose), or much delayed and infrequently, in later articles. Even though every peer reviewed piece has received lengthy discussion before publication, the readers don’t get to see the debates or read the reflections of expert reviewers. The privacy can allow the reviewer to write candidly, but the intellectual labor of writing reviews is a bit of a lost resource.

    I’d love to see a well supported, open access, open review anthropology journal. This sort of project could be extraordinarily successful in the long run, especially once that ‘small vanguard of digitally adept scholars’ mentioned in the NY Times article does the hard work of cracking through professionally, and can become the tenure reviewers for the next generation. I think it’s inevitable, but I have no idea what the time frame will be. Hope it’s soon.

  3. TILL STANDARDS, SUSTAINABILITY AND SCALABILITY ARE TESTED:
    PEER COMMENTARY IS SUPPLEMENT, NOT SUBSTITUTE, FOR PEER REVIEW

    See: Peer Review Reform: bit.ly/peer-review-reform

    Harnad, S. (1978) Inaugural Editorial. Behavioral and Brain Sciences 1(1).
    http://www.ecs.soton.ac.uk/~harnad/Temp/Kata/bbs.editorial.html

    Harnad, S. (ed.) (1982) Peer commentary on peer review: A case study in scientific quality control, New York: Cambridge University Press.

    Harnad, Stevan (1985) Rational disagreement in peer review. Science, Technology and Human Values, 10 p.55-62. cogprints.org/2128/

    Harnad, S. (1990) Scholarly Skywriting and the Prepublication Continuum of Scientific Inquiry Psychological Science 1: 342 – 343 (reprinted in Current Contents 45: 9-13, November 11 1991). cogprints.org/1581/

    Harnad, S. (1996) Implementing Peer Review on the Net: Scientific Quality Control in Scholarly Electronic Journals. In: Peek, R. & Newby, G. (Eds.) Scholarly Publishing: The Electronic Frontier. Cambridge MA: MIT Press. Pp 103-118. cogprints.org/1692/

    Harnad, S. (1997) Learned Inquiry and the Net: The Role of Peer Review, Peer Commentary and Copyright. Learned Publishing 11(4) 283-292. Short version appeared in 1997 in Antiquity 71: 1042-1048. Excerpts also appeared in the University of Toronto Bulletin: 51(6) P. 12. cogprints.org/1694/

    Harnad, S. (1998/2000/2004) The invisible hand of peer review. Nature [online] (5 Nov. 1998), Exploit Interactive 5 (2000): and in Shatz, B. (2004) (ed.) Peer Review: A Critical Inquiry. Rowland & Littlefield. Pp. 235-242. cogprints.org/1646/

    Harnad S. (2002) BBS Valedictory Editorial (2002) Behavioral and Brain Sciences 24 users.ecs.soton.ac.uk/harnad/Temp/bbs.valedict.html

    Harnad, S. (2003) PostGutenberg Peer Review the invariant essentials and the newfound efficiencies users.ecs.soton.ac.uk/harnad/Temp/peerev.pdf

    Harnad, S. (2009) The PostGutenberg Open Access Journal. In: Cope, B. & Phillips, A (Eds.) The Future of the Academic Journal. Chandos. eprints.ecs.soton.ac.uk/15617/

Leave a comment